Package: libvstruct2
libvstruct2YAML-like structured binary data representation language (partially based on Python). Goals:briefexpressive (capable to describe most contemporary file/data formats)human-readablefast (enough)easy to parseeasy to translate into Python parsersWARNING: very alpha release, many things are still broken / can change.Note: some automagical conversions happen when translating.vs2sources to.pysources to simplify
writing the definitions (to-be-documented). See the included file formats for examples.Vlad Topan (vtopan/gmail), Feb. 2020FormatYAML-like, mapping structures to- field:typelists or detailed field descriptions.-field:-type-key1:value1-key2:value2example:StructName:-field1:I4# 4-byte integer - this is ignored-field2:-a[100]-desc:some 4-byte integer field-offset:1234shorthands exist (seePFX_MAPinlibvstruct2/defparser.pyfor a full mapping), the [Fields][] section below for details:@ ...is equivalent to- offset: ...= ...is equivalent to- validator: ...(dynamic expression, exact value or collection)? ...is equivalent to- lookahead: ...(preview data, don't consume buffer)VStruct objectsEach VStruct object (structure or field) has the obvious attributes (.offset,.size,.value,.parent, etc.) and
some less obvious attributes and methods:.root: the top-level VStruct which initiated the parsing process.first_parent(type_or_typename): returns the first (grand)parent of the given typeFieldssyntax:<indent>- <fieldname>: <fieldtype>followed optionally by additional modifiersmost modifiers also have a short syntax, either as a modifier prefix or as a suffix to the<fieldtype>prefix example:- dyn_offset: <offset>is the same as@ <offset>suffix example:- field1: R[<length>]is the same as- field1: Rfollowed by- length: <length>modifiers (with prefixes/suffixes in parenthesis where available):prefixes:validator(=): after evaluation, test the resulting value with this expressioncan be alist/tuple,dict, callable or exact (immediate) valueshould_be(?=): same asvalidator, but instead of stopping the parser, only a warning is issueddyn_offset(@): dynamic offset - evaluated at runtime (seeDynExpr)formatter(~): used to format the value for display - a callable or adict(if value is not in dict,?is returned)hidden(.): don't display when displaying the structure (flag used for internal / irrelevant fields)description("..."): field descriptionbit_map({...}): masks mapping bits to nameslookahead(?): parsing the field won't consume from the buffer (the current offset is not changed)out_of_struct(*): the field does not follow from the current offset (this is set automatically bydyn_offset)required(!): flag which marks a conditional field as required (raisesParseErrorif missing)post_parse(;): function to be called after parsing for additional processing (receives an object instance)post_parse_value(:): function to be called after parsing to further process the value (receives the object instance, must return the new value)can also be a list of functions if the value is a sequence (only the values will be passed to the functions)suffixes:length([...]): number of elements (field becomes anarray, i.e.listof fields)exception: for string and regex types, thelengthmeans the number of characters!if the value starts with<, it is interpreted asmaxlen(maximum length), e.g. for NULL-terminated stringscondition(<...>):DynExprwhich, if evaluates to False, the field is considered "not present"no shortcut syntax (yet):update_offset: flag which causes an out-of-struct field to still update its parent's.offsetendianness: little ('<') or big ('>'), default: littlenote: this is inherited from parent structures if not defined in fieldsstop: alternative stop condition (DynExpr) for arrays (in this case,lengthmay be simplyTrue)second_pass: (only for out_of_struct nodes) parse only on a second passvirtual fields can be created with the syntax:- <fieldname> := <expression>(their.sizewill be 0, but the.offsetwill be correct)these are instances of theEvalbasic typeExpressionsDynExprmost values can be defined as dynamic expressions (to reference other fields which become available during parsing)automagically identified by trying to parse each expression as an immediate; on failure, a DynExpr is createdsyntax:$<field>extracts the value of<field>in the current (VStruct) structure (equivalent toself.<field>.value)@<field>the offset of<field>(self.<field>.offset):<field>the size of<field>(self.<field>.size)&<attr>extracts the value of an attribute of the current VStruct (equivalent toself.<attr>)parent objects can be referenced using^, e.g.:&^^==self.parent.parent$^Field1==self.parent.Field1.valuenotes:depending on the context,selfis usually the currentVStruct(e.g. when evaluatingconditions,lengths, etc.);
an exception exist though -selfis the current field when validating (validatorexpressions)Regex expressionsthe match is attempted from the current positionformat:<op>/<regex>/<flags><op>: what result to returndefault (no<op>field): bool (Trueif matched)@: match offset or -1:: match length<flags>: sequence of classic regex flags (sdotall,mmultiline,icase insensitive) and custom flags:f: attempt match in full file (default: 8k from current offset)Datatypesseelibvstruct2/basefields.pyfor the implementation of all the basic typesNumbersIntegersI1/I2/I4/I8: signed 1/2/4/8-byte integer (default endianness, little if none set)PK1..8: packed integer (DWARF3 LEB128) - TBDsignednessprefix:S/U(default: unsigned)endiannesssuffix:B: big endianL: little endianE: dynamic endianness (set by a conditional expression in the struct)default: little endianReal numbersF2/F4: floating-point 2/4-byte real numbers (float/double)Bit masks (flags)BM#(#is the number of bytes, e.g.BM2): flagsaccepts abit_map({ ... }) key which maps bits/bit groups to namesText & binary blobsa/u8/16/32/w/r: null-terminated ASCII / UTF 8/16/32 / widechar / raw (binary) string (C strings) - TBDAS[len]/WS[len]/R[len]: fixed/maximum-length ASCII / widechar / raw (binary) string (LPSTR/PWSTR)RegEx-based fieldsRX: basic (binary) regex field (the actual regex must be passed as the condition, e.g.- MyField: RX <^abc>).valuewill contain the groups tuple if more than one, a singlebytesotherwise.raw_matchwill contain the full (binary) matchT: string token (ignores whitespace) - multiple values separated by a comma (e.g.- MyField: T <a, b, cde>)SRX: string regex (UTF-8encoded)IS: integer string (textual integer);.valuewill be the actual numberKeysany type can be followed by any number of "keys" (indented- name: valuefields) with supplemental rules / informationFeedbackFeedback is welcome via email or as issues on the gitlab project.