Package: paradict
A Paradict config documentParadictStreamable multi-format serialization with schemaThis project is part of thePyrustic Open Ecosystem.Table of contentsOverviewParadict textual format: Why not JSON, YAML, or TOML ?Paradict binary format: Why not Protobuf, MessagePack, or CBOR ?Code snippets for everyday scenariosParadict datatypesData format specificationApplication programming interfaceTextual serializationBinary serializationType customizationContinuous data stream processingParadict schema for data validationMiscellaneousTesting and contributingInstallationOverviewParadictis a multi-formatserializationsolution for serializing and deserializing adictionarydata structure in bulk or in a streaming fashion.It comes with a data validation mechanism as well as other cool stuff, and its eponymous reference library is aPythonpackage available onPyPI.Read thebackstoryin thisHN discussion!A rich set of datatypesA Paradict dictionary can be populated with strings, binary data, integers, floats, complex numbers, booleans, dates, times,datetimes, comments, extension objects, and grids (matrices).Although Paradict's root data structure is a dictionary, lists, sets, and dictionaries can be nested within it at arbitrary depth.An extension mechanismParadict has an extension mechanism that works with two components:extension object: dictionary-based structures defined in Paradict data (in textual or binary format).object builder: Python callable (passed to deserializer) that takes an extension object as input, consumes its contents, builds and returns a new Python object.A multi-format solutionParadict offers binary and textual representations for a compatible arbitrary dictionary data structure.The human-readable format has two modes, adata-modefor bidirectional mapping to binary format, and aconfig-mode, with lighter syntax, suitable forconfiguration files.A validation mechanismData validation is performed against a schema which is itself just another dictionary. The schema can be defined in a file with an arbitrary data format (Paradict, JSON, etc.) or programmatically.Basically, a schema describes the expected keys in the target dictionary and the expected data types of their values. When defined programmatically, the schema allows the programmer to validate the target dictionary with arbitrary rules by incorporating checkercallbacks.An intuitive APIThe libraryAPIis designed to be simple to understand, intuitive and powerful. There are four fundamental classes:Encoder,Decoder,Packer, andUnpacker, which serialize and deserialize data iteratively.On top of these classes, four functions namelyencode,decode,pack, andunpackdo the same thing but in bulk.Then there are additional classes and functions to perform various tasks such asTypeRefclass for customizing types,ConfigFileclass for configuration files,load, anddumpfunctions for reading and writing Paradict binary files, etc.And more...There's more to say about Paradict that can't fit in this Overview section.In the following sections, we'll dig deeper into Paradict, but first, why notJSON,YAML,TOML,Protobuf,MessagePack, orCBOR?Back to topParadict textual format: Why not JSON, YAML, or TOML ?With its textual format, Paradict is de-facto alternative toJSON,YAML, andTOML. Although these three formats are all human-readable, they serve different purposes.For example, TOML is specifically designed for configuration files while JSON is used as a data interchange format.Having two modes (data-modeandconfig-mode) for its textual format makes Paradict an interesting solution that targets the different purposes of JSON, YAML, and TOML.Paradict, while offering a binary representation of its textual format, does also reject complexity and ambiguity as it can be found on YAML, has a great extension mechanism and a rich set of datatypes.Back to topParadict binary format: Why not Protobuf, MessagePack, or CBOR ?With its binary format, Paradict is de-facto alternative toProtobuf,MessagePack, andCBOR. However, choosing a binary format requires careful consideration as its strengths and weaknesses are not as readily discernible as in the case of a textual format.Therefore, this section can be expected to offer comprehensive benchmarking and comparison details on different serialization solutions.Nonetheless, given the potential bias of benchmarking toward a desired outcome, let us only point out that, unlike others, Paradict provides bidirectional mapping between its textual and binary formats.The surge inLLMadoption is a reminder that people value advanced machine interfaces and intuitive data representation, despite extra compute costs.Back to topCode snippets for everyday scenariosFollowing are working code snippets for everyday scenarios.Binary representation of dataPack and unpack:fromparadictimportpack,unpackmy_dict={0:42}# serialize my_dictbin_data=pack(my_dict)# testassertmy_dict==unpack(bin_data)Read and write a file:fromdatetimeimportdatetimefromparadictimportload,dumppath="/home/alex/test/user_card.bin"user_card={"name":"alex","id":42,"group":"admin","birthday":datetime(2020,1,1,4,20,59)}# serialize user_card then dump it into the filedump(user_card,path)# deserialize user_card from the filedata=load(path)# testassertuser_card==dataThe code snippet above will serialize theuser_carddictionary then dump it into theuser_card.binfile. The file would contain 43 bytes as following:fromparadictimportstringify_binpath="/home/alex/test/user_card.bin"withopen(path,"rb")asfile:data=file.read()print(stringify_bin(data))Output:\x01\x44\x6e\x61\x6d\x65\x44\x61\x6c\x65\x78\x42\x69\x64\xc5\x45\x67\x72\x6f\x75\x70\x45\x61\x64\x6d\x69\x6e\x48\x62\x69\x72\x74\x68\x64\x61\x79\x18\x9b\x2e\x2b\x3d\xa4\xffTextual representation of dataEncode and decode:fromparadictimportencode,decodemy_dict={0:42}# serialize my_dicttxt_data=encode(my_dict)# testassertmy_dict==decode(txt_data)Working with config filesCreate and interact with a configuration file:fromdatetimeimportdatetimefromparadictimportConfigFile,boxpath="/home/alex/test/app_settings.dict"# data for the 'user' sectionuser_card={"name":"alex","id":42,"group":"admin","birthday":datetime(2020,1,1,4,20,59)}# data for the 'gui' sectiongui_config={box.CommentID():box.Comment("Exotic fonts are banned !"),"font_family":"Arial","background":"black","dimensions":{"width":42.0,"height":3.14}}confile=ConfigFile(path)confile.set("user",user_card)confile.set("gui",gui_config)# few hours later...confile=ConfigFile(path)# testassertuser_card==confile.get("user")The code snippet above will create a config file then fill it with:[user]
name = "alex"
id = 42
group = "admin"
birthday = 2020-01-01T04:20:59

[gui]
# Exotic fonts are banned !
font_family = "Arial"
background = "black"
dimensions = (dict)
    width = 42.0
    height = 3.14Back to topParadict datatypesFollowing are Paradict datatypes for both textual and binary formats:dict: dictionary data structurelist: list data structureset: set data structureobj: object type for extensiongrid: grid data structure for storing matrix-like databool: boolean type (true and false)str: string type with unicode escape sequences supportraw: raw string without unicode escape sequences supportcomment: comment datatypebin: binary datatypeint: integer datatypefloat: float datatypecomplex: complex numberdatetime:ISO 8601datetime (with time offsets)date: ISO 8601 datetime: ISO 8601 time (with time offsets)Paradict supportsnullfor representing the intentional absence of any value.For the dictionary data structure, Paradict allows keys to be either strings or numbers. However, in the config mode of the textual format, keys should only be alphanumeric strings with underscores or hyphens.Paradict allows ordinary and raw strings, integers, and float numbers to span over multiple lines when they are tagged with(text),(raw),(int), and(float), respectively.Back to topData format specificationThis section is just an overview of the binary and the textual Paradict formats. For more information, consulttxt_paradict_spec.mdandbin_paradict_spec.md.Textual formatAt the high level of the textual representation is themessagewhich represents a dictionary data structure and at the low level is thelineof text. A line of text can represent either complete data, such as a number, or a portion of some data that spans multiple lines, such as a multiline string.For human readability, data expected to span multiple lines is first introduced with atag(the data type in parentheses) under which the data is placed with the correct number of4-space indents.The format comes with two modes, the data mode and the config mode. These modes differ based on the data type of dictionary keys and the character utilized to separate each key from its corresponding value.Data modeThe data mode formally represents data (bidirectional mapping to binary format). It allows strings and numbers as keys and use a colon as separator between a key and its value.# this is a comment
"my key": "Hello World"Config modeThe config mode is only for configuration files. It only allows strings as key, removing the need to surround them with quotes, and also uses the equal sign as separator between a key and its value.# this is a commentmy_key="Hello World"Read the full specification intxt_paradict_spec.md!Binary formatAt the high level of thebinaryrepresentation is themessagewhich represents adictionarydata structure and at the low level is thedatumwhich is often a 2-tuple composed of atagand itspayloadwhich may be non-existent.The binary format is designed from scratch, thus each datatype benefited from a scrupulous attention in order to have a compact and coherent binary representation.Read the full specification inbin_paradict_spec.md!Back to topApplication programming interfaceThe API exposes four foundational classes, Encoder, Decoder, Packer, and Unpacker, that serialize and deserialize data iteratively.On top of these classes, four functions, encode, decode, pack, and unpack, do the same thing but in bulk.Then there are additional classes and functions to do various stuff such as the TypeRef class for types customization, the ConfigFile class for configuration files, load and dump functions for reading and writing binary Paradict file, etc.Note that this section is just an overview of the API, thus it doesn't replace the module documentation.Exploremodule documentationTextual serializationEncoder and Decoder are the foundation classes for serializing and deserializing data. These classes process data iteratively. On top of these classes, two functions, encode and decode, do the same thing but in bulk.Three additional classes which are Document, FileDoc, and ConfigFile, offer a document abstraction for interacting with serialized data. A document can be divided into sections, each having a header and a dictionary body.Under the hood, document abstraction usesBraqUsing the Encoder classThe Encoder constructor acceptsmode, type_ref, skip_comments and skip_bin_data as arguments.Theencodemethod of this class takes as input a Python dictionary, then iteratively serialize it, yielding a line after another.fromparadictimportEncoderdata={"id":42,"name":"alex"}encoder=Encoder()# mode=const.DATA_MODElines=list()forrinencoder.encode(data):lines.append(r)print("\n".join(lines))Output:"id": 42
"name": "alex"The same code but with constructor parametermodeset toconst.CONFIG_MODEwould output:id = 42
name = "alex"Using the Decoder classThe Decoder constructor acceptstype_ref,receiver,obj_builderandskip_commentsas arguments.Thefeedmethod of this class takes as input a multiline string that represent the data to deserialize. This string can be fed up to the deserializer, line by line.fromparadictimportDecodertext='id = 42\nname = "alex"'decoder=Decoder()decoder.feed(text)ifdecoder.queue.buffer:decoder.feed("\n")decoder.feed("===\n")# end of streamdata=decoder.dataprint(type(data))print(data)Output:<class 'dict'>
{'id': 42, 'name': 'alex'}Using the encode functionTheencodefunction acceptsdata,mode,type_ref,skip_comments, andskip_bin_dataas arguments.fromparadictimportencode,constdata={"id":42,"name":"alex"}# DATA MODEr=encode(data)# mode==const.DATA_MODEprint("DATA MODE")print(r)# CONFIG MODEr=encode(data,mode=const.CONFIG_MODE)print("\nCONFIG MODE")print(r)Output:DATA MODE
"id": 42
"name": "alex"

CONFIG MODE
id = 42
name = "alex"Using the decode functionThedecodefunction acceptstype_ref,receiver,obj_builder, andskip_commentsas arguments.fromparadictimportdecode# for the sake of the example,# the 'id' key-value line follows the DATA mode# and the 'name' key-value line follows the CONFIG modedata="""\"id": 42name = "alex""""r=decode(data)print(r)Output:{'id': 42, 'name': 'alex'}Using the document abstractionThe ConfigFile class is based on the FileDoc class, itself inheriting from the Document class.Using the Document classThe Document class offers a document abstraction made of sections. Thus, viagetandsetmethods, one might read and write the body of a specific section by providing its header.The Document class has a built-in data validation mechanism as it accepts a schema as optional constructor argument.Following are methods exposed by the Document class:get,set,check,render,load_schema,validate,load_from,save_to,remove, andclear.fromparadictimportDocumentinit_text="""\id = 42name = "alex"[misc]project = "paradict"is_open_source = True"""doc=Document(init_text)# the default header argument is an empty stringassertdoc.get("")=={"id":42,"name":"alex"}# get the misc sectionassertdoc.get("misc")=={"project":"paradict","is_open_source":True}# edit the body of the misc sectiondoc.set("misc",{"pi":3.14})assertdoc.get("misc")=={"pi":3.14}Using the FileDoc classThe FileDoc class works as the Document class, except that it is linked to an actual file. This class also has additional methods which areupdate,load, andsave.fromparadictimportFileDocpath="/home/alex/file_doc.dict"file_doc=FileDoc(path)file_doc.set("",{"key":"value"})Using the ConfigFile classThe ConfigFile class works like the FileDoc class, except that it is linked to an actual configuration file, so its encoding mode is implicitly set toCONFIG_MODE.fromparadictimportConfigFilepath="/home/alex/app_config.dict"file_doc=ConfigFile(path)file_doc.get("section",skip_comments=False)# by default, comments are skippedMiscellaneous functionsUnder the hood, theDeserializerclass uses a public function for splitting a key-value line into three parts:the key,the value,and the separator character.fromparadictimportsplit_kvkey_val="my_key = 'my value'"info=split_kv(key_val)# info is a namedtuple containing# the key, the value, the separator char# which is either a colon ':', or an# equal '=', and also the mode which is either# const.CONFIG_MODE or const.DATA_MODEkey,val,sep,mode=infoBinary serializationPacker and Unpacker are the foundation classes for serializing and deserializing data. These classes process data iteratively and on top of them, two functions, pack and unpack, do the same thing but in bulk.Two additional functions, load and dump offer to read and write binary files.Using the Packer classThe Packer constructor accepts type_ref, and skip_comments as arguments.Thepackmethod of this class takes as input a Python dictionary, then iteratively serialize it, yielding a binary datum (or part of it) after another.fromparadictimportPacker,stringify_bindata={"id":42,"name":"alex"}packer=Packer()lines=list()buffer=bytearray()fordinpacker.pack(data):buffer.extend(d)print(stringify_bin(buffer))Output:\x01\x42\x69\x64\xc5\x44\x6e\x61\x6d\x65\x44\x61\x6c\x65\x78\xffUsing the Unpacker classThe Unpacker constructor acceptstype_ref,receiver,obj_builderandskip_commentsas arguments.Thefeedmethod of this class takes as input some binary data that represent the data to deserialize. This binary data can be fed up to the deserializer, by small amount of chunks.fromparadictimportpack,Unpackerdata={"id":42,"name":"alex"}d=pack(data)unpacker=Unpacker()unpacker.feed(d)assertunpacker.data==dataUsing the pack functionThepackfunction acceptsdata,type_ref, andskip_commentsas arguments.fromparadictimportpack,stringify_bindata={"id":42,"name":"alex"}# DATA MODEr=pack(data)print(stringify_bin(r))Output:\x01\x42\x69\x64\xc5\x44\x6e\x61\x6d\x65\x44\x61\x6c\x65\x78\xffUsing the unpack functionTheunpackfunction acceptsraw,type_ref,receiver,obj_builder, andskip_commentsas arguments.fromparadictimportpack,unpackdata={"id":42,"name":"alex"}d=pack(data)r=unpack(d)assertdata==rLoad and dumpfromparadictimportdump,loadpath="/home/alex/user_card.bin"data={"id":42,"name":"alex"}# Serialize and write data to user_card.bindump(data,path)# Read and deserialize datar=load(path)# testassertdata==rMiscellaneous functionsThe library exposes some public miscellaneous functions to play with binary data:forge_binfunction to generate a bytearray forged with the provided arguments which can be of bytes, byterarrays, integers,stringify_binfunction that returns the hexadecimal string representation of some binary data given as argument.fromparadictimportstringify_bin,forge_binargs=(b'\x01',b'\x02',None,3)r=forge_bin(*args)print(stringify_bin(r))Output:\x01\x02\x03Type customizationThe classes and functions for (de)serializing data, all accept an instance ofTypeRef.TypeRefis the class that is at the core the type customization mechanism.For example, one might want to only use Python's OrderedDict instead of the regular dict:fromcollectionsimportOrderedDictfromparadictimportTypeRef,decodedata="""\pi = 3.14user = (dict)id = 42name = "alex""""type_ref=TypeRef(dict_type=OrderedDict)r=decode(data,type_ref=type_ref)asserttype(r)isOrderedDictasserttype(r["user"])isOrderedDictassertr=={"pi":3.14,"user":{"id":42,"name":"alex"}}Also withTypeRef, one couldadaptsome exotic datatype, thus it will
conform with Python datatypes allowed for serialization:fromparadictimportTypeRef,encodeclassCapitalizedString(str):# an exotic typepasstype_adapter=lambdas:s.capitalize()adapters={CapitalizedString:type_adapter}type_ref=TypeRef(adapters=adapters)data={"name":CapitalizedString("alex")}r=encode(data,type_ref=type_ref)print(r)Output:"name": "Alex"Back to topContinuous data stream processingParadict supports both textual and binary continuous data stream processing.Textual streamFollowing is a heavily commented code snippet for performing continuous data stream processing:fromparadict.serializer.encoderimportEncoderfromparadict.deserializer.decoderimportDecoder# This stream is made of messages# Each message is a dictionary that serves as envelopestream=[{0:"a"},{0:"b"},{0:"c"}]# Result will hold the unpacked messagesresult=list()# instantiate encoder and decoderencoder=Encoder()# the receiver takes as argument the reference to the decoderdecoder=Decoder(receiver=lambdaref:result.append(ref.data))# iterate over the stream to pack each message into datums# that will feed the decoder which will call the receiver# after each complete unpacking of a message.# The decoder holds a reference to the latest# unpacked message via the "decoder.data" propertyfori,msginenumerate(stream):forlineinencoder.encode(msg):decoder.feed(line+"\n")decoder.feed("===\n")# check if datum is well unpackedassertmsg==decoder.data# decoder.data holds unpacked data# check if the original stream contents is mirrored in# the result variableassertstream==resultBinary streamFollowing is a heavily commented code snippet for performing continuous data stream processing:fromparadict.serializer.packerimportPackerfromparadict.deserializer.unpackerimportUnpacker# This stream is made of messages# Each message is a dictionary that serves as envelopestream=[{0:"a"},{0:"b"},{0:"c"}]# Result will hold the unpacked messagesresult=list()# instantiate packer and unpackerpacker=Packer()# the receiver takes as argument the reference to the unpackerunpacker=Unpacker(receiver=lambdaref:result.append(ref.data))# iterate over the stream to pack each message into datums# that will feed the unpacker which will call the receiver# after each complete unpacking of a message.# The unpacker holds a reference to the latest# unpacked message via the "unpacker.data" propertyfori,msginenumerate(stream):fordatuminpacker.pack(msg):unpacker.feed(datum)# check if datum is well unpackedassertmsg==unpacker.data# unpacker.data holds unpacked data# check if the original stream contents is mirrored in# the result variableassertstream==resultBack to topParadict schema for data validationA Paradict schema is a dictionary containing specs for data validation.A spec is either simply a string that represents an expected data type, or aSpecobject that can contain a checking function for complex validation.Supported spec strings are:dict,list,set,obj,bin,bin,bool,complex,date,datetime,float,grid,int,str,timeCode snippet:fromparadictimportvalidatefromparadict.validatorimportSpec# datadata={"id":42,"name":"alex","books":["book 1","book 2"]}# schemaschema={"id":Spec("int",lambdax:40<x<50),"name":"str","books":["str"]}assertvalidate(data,schema)isTrueBack to topMiscellaneousThe beautiful cover image is generated withCarbon.Back to topTesting and contributingFeel free toopen an issueto report a bug, suggest some changes, show some useful code snippets, or discuss anything related to this project. You can also directly emailme.Setup your development environmentFollowing are instructions to setup your development environment# create and activate a virtual environmentpython-mvenvvenvsourcevenv/bin/activate# clone the project then change into its directorygitclonehttps://github.com/pyrustic/paradict.gitcdparadict# install the package locally (editable mode)pipinstall-e.# run testspython-munittestdiscover-f-stests-t.# deactivate the virtual environmentdeactivateBack to topInstallationParadictiscross-platform. It is built onUbuntuand should work onPython 3.5ornewer.Create and activate a virtual environmentpython-mvenvvenvsourcevenv/bin/activateInstall for the first timepipinstallparadictUpgrade the packagepipinstallparadict--upgrade--upgrade-strategyeagerDeactivate the virtual environmentdeactivateBack to topAbout the authorHello world, I'm Alex, a tech enthusiast ! Feel free to get in touch withme!Back to top