Package: cstruct2
cstruct2Version: 1.0.2 (Beta)cstruct2is a Python library that hopes to simplify and ease the experience of working with C-style packed binary structures. That is, if you were to directly write the memory of a packed (not padded, there's a difference!) structure in C to a file, this library would be able to hasslelessly read them. There exists a myriad of other means to do this in Python, such as reading each byte by hand or using Python's nativestructlibrary, but these are clunky and hard to read solutions.cstruct2provides a metaprogramming interface, wherein you define a class decorated bycstruct2that defines each field to be read, in an intelligent way.The reading of fields of a certain type and explicit width is a given, butcstruct2allows you to base a field's length off of the value of another previously read field, making code easier to write and read. Not only that, you can tell acstruct2decorated object to read adifferentfield based off of the value of a previously read field, which is highly convenient if you're trying to parse a dusty network protocol (SOCKS5 comes to mind). Of course, it wouldn't be complete without allowing you to recursively read nestedcstruct2objects within one another, whichcstruct2allows you to do! There's more than that, of course, but those are the main highlights and superiorities ofcstruct2.Basic UsageEach field is a member of a decorated class, where the type of the data item is given as a type annotation and the value of that data item is its size, in most cases. For example, if we wanted to define an objectMyStructurewhich has an integer 4 bytes in length--cstruct2assumes Little Endian ordered integers--and an ASCII string 32 bytes in length:from cstruct2.cstruct2 import cstruct2
    
@cstruct2
class MyStructure:
    my_number: int = 4
    my_string: str = 32Everycstruct2derived structure will then immediately be instantiated upon invoking its name, where a.from_stream()member will be exposed. One will call this method with a Pythonfile-like object, most importantly an object with theread()method for reading bytes in a stream. This can be from a socket, from an actual file, from a BytesIO or StringIO object, or anything really. Upon invoking this member on such an object, a dictionary will be returned with the values read. In the example above, the following dictionary is within the realm of possibility upon the.from_stream()member being called:with open("file.bin", "rb") as fp:
    print(MyStructure.from_stream(fp))

-->

{
	"my_number": 256,
	"my_string": "This is a string."
}Primitive Types and Variable LengthBecause we are dabbling in the wonders of Python metaprogramming, we've kind of created our own mini programming language.cstruct2has a few primitive types, which are just simple standard data types that have a length and can be read in.cstruct2currently supports the following primitive types, which are denoted on each field as a Python type annotation:str- Any kind of string, could be encoded in a myriad of different ways.int- Any kind of integer, which can be encoded in either Little or Big Endian, with differing widths.float- Any kind of floating point number, with widths 4 or 8 bytes and differing endianness.bytes- An arbitrary number of bytes, where a Pythonbytesobjects will be returned to store the bytes.(Future) (For writing only)anyfield: allows any type to be written to a stream with the structure, with the field being provided at runtime. When declaring this in the structure, it must be initialized toNone.cstruct2will throw an exception if this field is in a structure that is reading from a stream. When writing the structure to a stream, this field's value in the data dictionary must be a tuple of(field tuple, value).Already, if you've been paying attention, we've encountered a problem. With all of the variations described above, how can we tellcstruct2about all of them, with one value in the object field declaration? If we want to get more specific about our field, we will have to make its value a tuple, a tuple of more specific information in addition to the field's width. For example, if we want to read in an integer worth 2 bytes but encoded as Big Endian, we'd have to declare:@cstruct2
class MyStructure:
    number: int = ("big", 2)Now the question is, what are the available options for settings in tuples, and what are their orders, for each primitive data type? Below, we will list the different settings, exceptions, and orders for each data type, when using them in tuple-form:str: (length,encoding,wrapper)Theencodingfield may be any string encoding value that Python uses when decoding strings (e.g., "utf-8" or "ascii").Iflengthis "null"--and this is really important--then a null-terminated string will be read in. That is,cstruct2will keep reading bytes until it gets a '\0' byte.Iflengthis "pascal", then the string's length will be determined by the first byte that precedes it, as in a Pascal-style string.(Future): Iflengthis "pascal16" or "pascal32", then the same as regular "pascal" above, but with 16 bit and 32 bit lengths.int: (endianness,length,wrapper)endiannesscan be explicitly declared by using "big" or "little", but "host" and "network" are both options, which are relative endianness options.float: (endianness,length,wrapper)endiannesshas the same string enumerations as above.bytes: (length,wrapper)(Future) (For writing only)anyfield:Structure value required to be NoneYou may have noticed awrapperfield at the end of every tuple, along with how it was left unexplained in each bullet-point. Thewrapperfield allows you to pass in an object or a function that will receive the read in field and will convert it to another object or datatype afterwards, when it finally puts it into the resulting dictionary, if in structure reading mode. For example, if we want to read in astrbut then want to cast it to our customUserclass:class User:
    def __init__(self, usr: str):
	    self.usr = usr

@cstruct2
class MyStructure:
    username: str = ("null", User)Pretty cool. But what if your C structure first sends auint16_tdescribing the number of bytes that the next string it sends will be? When parsing that manually, you can very easily become confused. However,cstruct2allows you to base the length of a later field on the currently read value of a prior field, by simply referencing that field as a string when you pass in astras a length argument. For example:@cstruct2
class MyStructure:
    len: int = ("little", 2)
    string: str = "len"This is the power of variable/derived lengths incstruct2--and it's simply amazing. Anywhere a field is supposed to have a length or a width, you can simply pass in a string that references a previously read field, PROVIDED that the field is an integer type within thecstruct2decorated structure. It doesn't matter if you cast the field later on to anintby cleverly using wrappers, itmustbe anintfield within the structure.ArraysOf course, no structure parsing library is complete without the ability to read repeated elements, based off of another field's value. Very commonly are we asked to read innitems when dealing with such structures. Simply, arrays incstruct2are declared by setting the field's value to that of a list, where the length of the array is the first element, then the settings of the underlying element(s) in the second and onward elements. For example:class User:
    def __init__(self, usr: str):
	    self.usr = usr
	    
@cstruct2
class MyStructure:
    no_names: int = 4
    names: str = [
	    "no_names",
		16,
		User
    ]This structure will read inno_namesnumber of 16 byte strings, which will then all be casted toUserobjects, since that is what the wrapper was set to in thestrfield's options.The resulting dictionary may look something like this:{
    "no_names": 3,
    "names": [
	    <User obj blah blah 1>,
	    <User obj blah blah 2>,
	    <User obj blah blah 3>
    ]
}If one wants to wrap the resulting array into something, then the last argument of the list describing the array should be a callable type which then returns the wrapper, in the same way any other field can be wrapped.Switch FieldsIt is also very common for the field we read to be different based off of the value of another previously read field. If the SOCKS5 protocol tells us the address type can be IPv4, IPv6, or a string domain, we need to have some variability in our code. This is exactly what a switch field allows us to do.A switch field is made by declaring a field to be that of aswitchtype, then the value being a switch object which has two instantiation arguments:dependent_value: str, the previously read field we base our decisions ondecisions: dict, a map of what each value as read fromdependent_valueshould be as a anothercstruct2field.Due to the nature of the declared switch statement, the field type cannot be given as a type annotation. Instead, it must be given as the first argument of a tuple describing each possible field in a decision mapping. Of course, the last argument of the tuple for each decision field would describe a wrapper.Of course, with things like these, it's best to explain additionally with just an example:from cstruct2.cstruct2 import cstruct2, switch

@cstruct2
class MyStructure:
    usr_type: int = 1
    user: switch = switch(
	    "usr_type", {
		    0: (int, "little", 4),
			1: (str, "null"),
			2: (int, [3, "big", 8])
		}
	)The structure is saying: ifusr_typeis equal to 0, then read in a Little Endian 4 byte integer and place that intouserwhen we're done; ifusr_typeis equal to 1, then read in a null terminated string; finally, ifusr_typeis equal to 2, then read in an array of ints 2 elements in length, where each integer is a Big Endian 8 byte integer. Wow, would that be a nightmare to manually write!Nestedcstruct2structuresAs with C structures, we can have nestedcstruct2structures, where the result will be placed in a subobject in the main dictionary that parsing would yield. The field's type annotation for a nestedcstruct2structure must be that ofstructure, with the value equal to thecstruct2object representing that nested structure--and if you use a tuple, the last argument of that tuple would be some wrapper which would be fed the corresponding read dictionary of the nested structure. An example only will be provided, because it's crystal clear how it works:from cstruct2.cstruct2 import cstruct2, structure

@cstruct2
class UserStructure:
    id: int = 4
    username_len: int = ("big", 2)
    username: str = "username_len"

@cstruct2
class MainStructure:
    code: int = 1
    users_len: int = 2
    users: structure = [
	    "users_len",
	    UserStructure
    ]The resulting dictionary may look like something like this:{
	"code": 27272727,
	"users_len": 2,
	"users": [
		{
			"id": 1,
			"username_len": 5,
			"username": "hello"
		},
		{
			"id": 2,
			"username_len": 6,
			"username": "hello2"
		}
    ]
}It's breathtaking, if you've ever toiled away parsing packed binary structures manually. The amount of bugs, the amount of headaches, and the amount of silly mistakes...  Let's write some C code to compare!struct __attribute__((__packed__)) UserStructure {
	uint64_t id;
	uint16_t length;
};

struct __attribute__((__packed__)) MainStructure {
	uint8_t code;
	uint16_t users_len;
};

...

int main(void) {
	FILE *fp = fopen("file.bin", "rb");
	
	struct MainStructure mstruct;
	fread(&mstruct, 1, sizeof(mstruct), fp);
	for (uint16_t i = 0; i < mstruct.users_len; i++) {
		struct UserStructure ustruct;
		fread(&ustruct, 1, sizeof(ustruct), fp);
		uint16_t little_len = ntohs(ustruct.username_len);
		char *username = (char*) malloc(ustruct.little_len + 1);
		fread(username, 1, ustruct.little_len, fp);
		username[ustruct.little_len] = '\0';
		
		printf("[%d]: (%d) %s", i, ustruct.id, username);
		free(username);
	}

	fclose(fp);
	return 0;
}The manual Python version, if you're not usingstruct, is not much better (and in fact, can be longer!).Writing StructuresIt's natural to also want to write these structures back to a stream or to some other output, so naturallycstruct2supports it. Writing structures isn't scattered throughout the previous sections because it does not differ much from regularcstruct2structure reading mode. There are some minute differences, but they are minute enough to be all under one inclusive section.Acstruct2parsed structure will have theto_stream()andto_bytes()methods that will write the structure to either a stream or will write it to bytes which will be returned. In both of those methods, a dictionary containing the values for each defined field must be passed as the first argument. Obviously,to_stream()then needs the stream as its second argument. Let's give an example:@cstruct2
class MyStructure:
	age: int = ("little", 4)
	name: str = "null"
	things: int = [3, "little", 2]

with open("out.bin", "wb") as fp:
	MyStructure.to_stream({
		"age": 32,
		"name": "Test Person",
		"things": [2, 3, 5]
    }, fp)This will write the values to the corresponding fields to the stream provided, which points to a file. Because of how the structure was defined,agewill be written as a 4 byte integer in Little Endian order to the stream. Similarly,namewill be written as a null terminated string to the stream... you get the picture now.Auxiliary Componentscstruct2provides some wrapper classes and helper functions in its library, most notablycstruct2.cstruct2.SocketWrapper. If you want to read or write a structure from/to a socket, you will need to construct aSocketWrapperon that socket first, before utilizing it withcstruct2. For example, if I open a socket as a client, and I wish to utilize it withcstruct2:from cstruct2.cstruct2 import cstruct2
from cstruct2.cstruct2_utils import SocketWrapper

@cstruct2
class Handshake:
	id: int = ("little", 4)
	username: str = "pascal"
	password: str = "pascal"

@cstruct2
class Response:
	last_login: int = ("little", 8)
	messages_len: int = ("little", 2)
	messages: str = ["messages_len", "null"]

sock = socket.socket()
sock.connect(("192.168.0.3", 8080))
sock_wrapper = SocketWrapper(sock)

Handshake.to_stream({
	"id": 321,
	"username": "admin",
	"password": "123"
}, sock_wrapper)

response: dict = Response.from_stream(sock_wrapper)
for msg in response["messages"]:
	print(msg)

sock.close()A variety of examples are provided in this repository, such as an example SOCKS5 proxy server created withcstruct2and a C program that writes packed binary structures to a file, whichcstruct2will then read and parse.MiscellaneousA list of changes to this library can be seen through the CHANGELOG	file in this repository. A list of things that need to get done can be seen through the TODO file that is also in this repository. The source code to this library is quite messy and inefficient at the moment as well, as a heads up. If you have any questions, complaints, or suggestions, feel free to make issues on this repository or email me atarner@usa.com.