Package: py-datastruct
py-datastructThis is a (relatively) simple,pure-Python, no dependencylibrary, aiming to simplify parsing and building binary data structures. It usesdataclassesas its main container type, andstruct-compatible format specifiersfor writing field definitions.The way of composing structures is somewhat similar to (and inspired by)Construct. While probably not as powerful, it should give more flexibility and control over the data, as well asfull IDE type hinting.Installationpipinstallpy-datastructUsageThis simple example illustrates creating a 24-byte long structure, consisting of a 32-bit integer, an 8-byte 0xFF-filled padding, and a 12-bytebytesstring.fromhexdumpimporthexdumpfromdataclassesimportdataclassfromdatastructimportDataStructfromdatastruct.fieldsimportfield,padding@dataclassclassMyStruct(DataStruct):my_number:int=field("I",default=123)_1:...=padding(8)my_binary:bytes=field("12s")my_object=MyStruct(my_binary=b"Hello Python")print(my_object)# MyStruct(my_number=123, my_binary=b'Hello World!')my_object=MyStruct(my_number=5,my_binary=b"Hello World!")print(my_object)# MyStruct(my_number=5, my_binary=b'Hello World!')packed=my_object.pack()hexdump(packed)# 00000000: 05 00 00 00 FF FF FF FF  FF FF FF FF 48 65 6C 6C  ............Hell# 00000010: 6F 20 57 6F 72 6C 64 21                           o World!unpacked=MyStruct.unpack(packed)print(unpacked)# MyStruct(my_number=5, my_binary=b'Hello World!')print(my_object==unpacked)# TrueYou might also pass a stream (file/BytesIO/etc.) topack()andunpack(). Otherwise,pack()will create a BytesIO stream and return its contents after packing;unpack()will accept abytesobject as its parameter.pack()andunpack()also accept custom, keyword-only arguments, that are available in the Context, throughout the entire operation.ContextUpon starting a pack/unpack operation, aContextobject is created. The context is a container scoped to the currently processed structure. It's composed of the following main elements:all values of the current structure - when packing; during unpacking, it contains all values of fields that were already processes (the context "grows")all keyword arguments passed topack()/unpack()(for the root context only)all keyword arguments passed tosubfield()(for child contexts only)_: Context- reference to the parent object's context (only when nestingDataStructs)self: Any- the current datastruct - note that it's aDataStructsubclass when packing, and aContainerwhen unpackingG- global context - general-purpose container that is not scoped to the current structure (it's identical for nested structs)io: IO[bytes]- the stream being read from/written topacking: bool- whether current operation is packingunpacking: bool- whether current operation is unpackingroot: Context- context of the topmost structuretell: () -> int- function returning the current position in the streamseek: (offset: int, whence: int) -> int- function allowing to seek to an absolute offsetP- local context - general-purpose container that is different for each nested structconfig: Config- current DataStruct's configtell: () -> int- function returning the current position in the current structure (in bytes)seek: (offset: int, whence: int) -> int- function allowing to seek to an offset within the current structureskip: (length: int) -> int- function allowing to skiplengthbytesi: int- (forrepeat()fields only) index of the current item of the listitem: Any- (forrepeat()fields, inlast=lambda only) item processed right before evaluationself: Any- (packing only) value of the current fieldThe context is "general-purpose", meaning that the user can write custom values to it. All fields presented above can be accessed by lambda functions - see "Parameter evaluation".Parameter evaluationMost field parameters support pack/unpack-time evaluation (which means they can e.g. depend on previously read fields). Lambda expressions are then given the current context, and expected to return a simple value, that would be statically valid in this parameter.an_unpredictable_field:int=field(lambdactx:"I"ifrandint(1,10)%2==0else"H")Ellipsis - special valueA special value of typeEllipsis/...is used in the library, to indicate something not having a type or a value.It's not the same asNone.built()fields, for example, have...as value after creating the struct, but before packing it for the first time.Special fields (likepadding(), which don't have any value) must have...as their type hint.Variable-length fieldsThis is a simple example of using parameter evaluation to dynamically size abytesstring. Binary strings use the<len>sspecifier, which can be omitted (simpleintcan be used instead).@dataclassclassMyStruct(DataStruct):data_length:int=field("I")data:bytes=field(lambdactx:ctx.data_length)The user is still responsible for adjustingdata_lengthafter changingdata. Thebuilt()field comes in handy here:@dataclassclassMyStruct(DataStruct):data_length:int=built("I",lambdactx:len(ctx.data))data:bytes=field(lambdactx:ctx.data_length)When unpacking, thedata_lengthfield will be used to dynamically size thedatafield. When packing,data_lengthwill always be recalculated based on what's indata.Wrapper fields - storing a listLists are also iterables, likebytes, but they store a number of items of a specific type. Thus, therepeat()fieldwrapperhas to be used.Wrapper fieldssimply require calling them first with any used parameters, then with the "base" field.@dataclassclassMyStruct(DataStruct):item_count:int=built("H",lambdactx:len(ctx.items))# This creates a list of 16-bit integers.# The list is empty by default.items:List[int]=repeat(lambdactx:ctx.item_count)(field("H"))my_object=MyStruct()my_object.items=[0x5555,0x4444,0x3333,0x2222]my_object.item_count=1# this doesn't matter, as the field is rebuiltpacked=my_object.pack()hexdump(packed)# 00000000: 04 00 55 55 44 44 33 33  22 22Conditional fieldsThey're also wrapper fields - if the condition is not met, they act like as if the field didn't exist at all.@dataclassclassMyStruct(DataStruct):has_text:bool=field("?")text:str=cond(lambdactx:ctx.has_text)(field("8s",default=""))my_object=MyStruct.unpack(b"\x01HELOWRLD")print(my_object)# MyStruct(has_text=True, text='HELOWRLD')my_object=MyStruct.unpack(b"\x00")print(my_object)# MyStruct(has_text=False, text='')Switch fieldsSwitch fields are like more powerful conditional fields. The following example reads an 8/16/32-bit number, depending on the prefixing length byte. If the length is not supported, it reads the value asbytesinstead.number_length:int=field("B",default=1)number:Union[int,bytes]=switch(lambdactx:ctx.number_length)(_1=(int,field("B")),_2=(int,field("H")),_4=(int,field("I")),default=(bytes,field(lambdactx:ctx.number_length)),)The values on the left (_1,_2,_4) are thekeys. The key is picked depending on the key-lambda result (ctx.number_length). The value on the right is a tuple of the expected field type, and afield()specifier.Since it's not possible to pass just1as a keyword argument, integers are looked up prefixed with an underscore as well. Enums are additionally looked up by their name and value, and booleans are looked up bylowercasetrue/false.Note that you can pass (probably) any kind of field to the switch list.To be continuedLicenseMIT License

Copyright (c) 2023 Kuba Szczodrzyński

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.