Package: bytechomp
bytechompA pure python, declarative custom binary protocol parser & generator using dataclasses and type hinting.bytechompleverages Python's type hinting system at runtime to build binary protocol parsing schemas from dataclass implementations. Deserialization/Serialization of the binary data is now abstracted away bybytechomp, leaving you to work in the land of typed and structured data.Features:Pure PythonZero DependenciesUses native type-hinting & dataclassesSupports lower-precision numericsSupportsbytesfields of known lengthSupportslisttypes for repeated, continuous fields of known lengthSupports nested structuresSupports serialization of populated data structuresInstallationbytechompis a small, pure python library with zero dependencies. It can be installed via PyPI:pip install bytechompor Git for the latest unreleased code:pip install https://github.com/AndrewSpittlemeister/bytechomp.git@mainReader APITheReaderclass uses Python's built-ingenericsdetermine the dataclass used when parsing. This dataclass is defined by the user to mimic the binary protocol. Once instantiated, theReaderclass can be fedbytesand used to construct the dataclass when ready. There are various ways to accomplish this with theReaderclass:fromdataclassesimportdataclassfrombytechompimportReader@dataclassclassMyStruct:timestamp:floatidentity:int# instantiate a readerreader=Reader[MyStruct]().allocate()# add data to the internal bufferreader.feed(stream.read(512))# check if enough data is present to buildprint(reader.is_complete())# add via the bitshift methodreader<<stream.read(512)# check via bool magic methodprint(bool(reader))# combine alternative methodsifreader<<stream.read(512):# construct dataclassmy_struct=reader.build()# clear internal byte bufferreader.clear()# use the iterator APIsimulated_byte_iterator=[b"a"]*10formy_structinreader.iter(simulated_byte_iterator):print(my_struct)Serialization APISimilar to theReader, serialization of data is accomplished through defining dataclasses in the same manner.frombytechompimportserializemy_struct=MyStruct(1.1,15)serialized_struct:bytes=serialize(my_struct)Supported Type FieldsFields on the dataclasses can be integers, floats, strings, bytes, lists, or other dataclasses. Python-nativeintandfloatrepresent 64-bit variants. Other sizes can be imported frombytechomp:frombytechomp.datatypesimport(U8,# 8-bit unsigned integerU16,# 16-bit unsigned integerU32,# 32-bit unsigned integerU64,# 64-bit unsigned integerI8,# 8-bit signed integerI16,# 16-bit signed integerI32,# 32-bit signed integerI64,# 64-bit signed integerF16,# 16-bit floatF32,# 32-bit floatF64,# 64-bit float)Although these allow aReaderto parse a field of a custom size, the resulting value populated in a dataclass field will always be the python-nativesintorfloat.Repeated fields likebytesandlistrequire the use of Python'styping.Annotatedto allow defining a length.frombytechompimportAnnotated,dataclass# re-exported by bytechomp@dataclassclassMessage:name:Annotated[bytes,10]identity:Annotated[bytes,10]flags:Annotated[list[int],5]Finally,listfields can contain any other supported datatype, including other dataclass structures to handle complex, nested protocols.Byte OrderingByte default the byte-ordering is set to the machine's native format, but can be changed:frombytechompimportReader,ByteOrder,dataclass,serialize@dataclassclassMyStruct:timestamp:floatidentity:int# use native (the default)reader=Reader[MyStruct](ByteOrder.NATIVE).allocate()data=serialize(MyStruct(1.1,15),ByteOrder.NATIVE)# use little endianreader=Reader[MyStruct](ByteOrder.LITTLE).allocate()data=serialize(MyStruct(1.1,15),ByteOrder.LITTLE)# use big endianreader=Reader[MyStruct](ByteOrder.BIG).allocate()data=serialize(MyStruct(1.1,15),ByteOrder.BIG)A Longer ExamplefrombytechompimportReader,dataclass,Annotated,serializefrombytechomp.datatypesimportU16,F32@dataclassclassHeader:timestamp:float# native datatypes can be used when assuming full precisionmessage_count:int# similarly with 64-bit integersmessage_identity:U16# custom datatypes are available and will be cast to native when deserialized@dataclassclassBody:unique_id:Annotated[bytes,5]# use of typing.Annotated to denote lengthbalance:F32@dataclassclassMessage:header:Header# nested data structures are allowedbody:Body@dataclassclassMessageBundle:messages:Annotated[list[Message],8]# so are lists of data structures!defmain()->None:# build Reader object using the MessageBundle class as its generic argumentreader=Reader[MessageBundle]().allocate()withopen("my-binary-data-stream.dat","rb")asfp:while(data:=fp.read(4)):# feed stream into the readerreader.feed(data)# check if the structure has been saturated with enough dataifreader.is_complete():# parse the stream and create your typed data structure!msg_bundle=reader.build()print(msg_bundle)# re-serialize this dataprint(f"serialized data:{serialize(msg_bundle)}")Other ExamplesSeeparse-sqlite-header.pyfor an example of usingbytechompto read the header message of an sqlite file. A rough estimate of what this should result in can be foundhere.Seetcp-client-server.pyfor an example of usingbytechompto serialize/deserialize binary messages across a TCP connection.How does this work?While a binary stream is usually represented as a flat, continuous data,bytechompcan be used as a structural abstraction over this data. Therefore, if there was a message with the following structure for a message calledUserState:FieldTypeDescriptionuser_iduint64user's unique identitybalancefloat32user's balanceThe resulting translation to a dataclass would be the following:frombytechompimportReader,dataclassfrombytechomp.datatypesimportF32@dataclassclassUserState:user_id:intbalance:F32When parsing messages that contain other messages, you will need to be aware of how the embedded messages are contained and how the resulting memory layout will look for the container message as whole. Since the container message is still represented as one set of continuous bytes, nested classes in bytechomp are constructed using a depth first search of the contained fields in nested structures to build out a flattened parsing pattern for Python'sstructmodule.Consider the following structures:frombytechompimportReader,dataclass,Annotated# using re-export from within bytechompfrombytechomp.datatypesimportF32@dataclassclassUserState:user_id:intbalance:F32@dataclassclassTransaction:amount:F32sender:intreceiver:int@dataclassclassUser:user_state:UserStaterecent_transactions:Annotated[list[Transaction],3]TheUsermessage would correspond to the following memory layout:uint64, float32, float32, int64, int64, float32, int64, int64, float32, int64, int64Additional NotesThis package is based on a mostly undocumented feature in standard implementation of CPython. This is the ability to inspect the type information generic parameters via theself.__orig_class__.__args__structures. The information in this structure is only populated after initialization (hence the need for theallocate()method when instantiated aReaderobject). Should this behavior change in future versions of Python,bytechompwill adapt accordingly. For now, it will stay away from passing a type object as a argument to initialization because that just seems hacky.Future Improvements:Perhaps allowing for parameterized fields to reference previously declared fields (i.e. allowing a list of sizenwherenis the previous field)Allow declaring value restraints on fieldsMaking use of thetyping.Literalpython classAllow for enums to be generated for integer fields