Package: jesth
Jesth has triggered the creation of two spin-offs to push its ideas further:Paradict: streamable multi-format serialization with schema.Braq: the most obvious way to section a document.These projects therefore make Jesth obsolete.JesthFile with 1 anonymous top section, and 2 'user' sectionsJesthHuman-readable versatile data formatThis project is part of thePyrustic Open Ecosystem.InstallationDemoLatestModulesTable of contentsOverviewWhy not TOML, YAML, or JSON ?Why use Jesth ?DemoCode snippets for everyday scenariosAnatomy of a SectionBase sectionDict sectionSection familySpecs of the Dict SectionScalarsNull valueSingle-line stringsMulti-line stringsDate and timeEmbedded binary dataComments and whitespacesCollections (nested dict and list)Application programming interfaceJesth Schema for data validationAvailable implementationsMiscellaneousContributingInstallationOverviewJesth(pronounced/dʒest/), which stands forJust Extract Sections Then Hack'em, is a human-readable data format whose reference parser is an eponymous lightweightPythonlibrary available onPyPI.Read thebackstoryin thisHN discussion!Versatile formatA Jesthdocumentis made up ofsections, each defined by aheader(between two square brackets) and abodywhich is just lines of text.Instead of interpreting these lines upstream after extracting the sections, the Jesth parser lays low and lets the programmer decide which hack to apply to them.Because of this parsing policy, a single Jesth document can contain aneclectic set of sectionsrepresenting a poem, server configs, help text,promptsfor a chatbot,directed graph,ascii artwork, and more.Jesthis used bySharedfor data persistence and exchange. VisitShared!Shipped with a proper and extensively tested hackFor convenience, the Jesth library includes a proper and extensively testedhackto convert a compatible section to adictionarydata structure. In such a section that allows comments, one can encodestrings(single or multi-line),scalars(integer, float, decimal float, complex, booleans),dateandtime,nullvalue,binarydata and nestedcollections(list and dictionary).Therefore, Jesth can be used for configuration files, to create ascripting language, to act as a markup language to writedocstringsso that a tool can generate reference documentation after parsing them, etc.In the next few sections, we'll dive deep into Jesth, but first, why notTOML,YAML, orJSON?Why not TOML, YAML, or JSON ?Jesth offers a high level ofversatilitythat allows the programmer to freely decide what a section can contain and how to interpret its contents.Moreover, whenever the programmer decides to adopt a dictionary as a data structure to formalize some or all sections of a Jesth document, he/she is offered avery readable and strict syntaxto encode the data.The library exposes an intuitive API for loading and dumping Dict sections with an option to preserve comments and whitespaces.The following subsections will briefly compareTOML,YAML, andJSONtoJesth's dictionary data structure support.Why not TOML ?Jesth shares withTOMLthe use of square brackets to define sections (or tables in TOML terminology), therefore, among other formats, the TOML syntax is closest to Jesth. However, when dealing with nested structures, TOML becomes very confusing, revealing the limitations of its syntax.Besides intuitiveness and readability, the design decisions behind Jesthaccidentallycreated anunlimited reserved word pool. For example, the[[END]]header (note the double brackets on either side) is reserved as a document separator, allowing you to mark the end of a Jesth stream or store multiple Jesth documents in a single file.Why not YAML ?YAMLtightly embraces complexity and ambiguity, which are probably not useful for most programmers' needs. Jesth keeps it minimalist, unambiguous and very readable.Why not JSON ?JSONis great because it mimics the way we define data structures in source code. Therefore, it can be minified and is also a reasonable choice for machine-to-machine communication, but it cannot beat the readability of formats that natively support line breaks and/or indentation. In comparison to Jesth, JSON has fewer native data types and no native support for comments.Why use Jesth ?Jesth is ideal for configuration files, whether intended for use by a standard application or by a chatbot/AI. Personal notes can also benefit from being entered into a Jesth file, so they can be parsed later. As mentioned in theOverviewsection, Jesth was used to design ascripting languageand currently serves as the markup language fordocstringsin all projects that are part of thePyrustic ecosystem.Loading a Jesth document is stress-free because exceptions are only thrown when you try to convert a non-compatible section to a dictionary. But even that is the strict default behavior of the dictionary converter since you can customize the converter to string invalid values. This specific customization is as simple as this:value_converter.fallback_decoder = strThere's more to say, for example the Jesth Dict Section syntax is text editor/IDE-friendly, so it may benefit from supporting some cool features like the folding mechanism.DemoThe demo is a clonable repository containing a JesthFile and a Python script to load its contents.Open the Demo!Code snippets for everyday scenariosLoad a JesthFile:fromjesthimportreadpath="/home/alex/jesthfile"document=read(path)Get sections:...# get the "config" sectionconfig_section=document.get("config")# get the list of all "user" sectionsuser_sections=document.get_all("user")# show the body of a sectionprint(config_section.body)# list of lines# show the body as a text (concatenated lines)print(config_section.to_text())# update the body of a sectionnew_body=[r"# comment",r"server1 = '1.1.1.1'",r"port = 80"]config_section.update(new_body)Convert a compatible section into a Python Dict:...# won't raise an exception if the section can't be converted to dictdict_object=section.to_dict()# will raise an exception if the section can't be converted to dictdict_object=section.create_dict()# Note that `.create_dict` accepts `default` and `strict` arguments# If the body can be converted to a dict but is empty, you get `default`# Set True to `strict` to preserve comments and whitespaces# Note that `.to_dict` accepts `default`, `strict`, and `fallback`# If the body can't be converted to a dict, you get `fallback`Edit a document:...# append a new "user" section at the end of documentbody={"name":"alex","id":420}document.append("user",body)# body as a regular Python dict# insert a new "user" section at the beginning of documentbody=[r"# comment",r"name = 'alex'",r"id = 420"]document.insert(0,"user",body)# body as a list of linesSave a document:fromjesthimportwrite...# if the document is linked to a JesthFile, call its `save` methoddocument.save()# or save to a new pathdocument.save_to(path)# Under the hood, the Document class actually uses the `write` functionwrite(document,path)Render a document:fromjesthimportrender...# return the string representation of the data# this jesth text can be stored as it in a JesthFilejesth_text=document.render()# Under the hood, the Document class actually uses the `render` functionjesth_text=render(document)Anatomy of a SectionAsectionis made up of two parts: aheaderand abody. The header is defined on its own line, surrounded by two single brackets (opening and closing brackets respectively).The body is what is between two consecutive headers or between a header and the end of the document.Example of headers:[this is a header]

[this_is_another_header]

\[this is not a header at all]

[[this is very illegal]]

\[[this is not illegal at all but still not a header]]Note:A header may be empty and referenced from the source code with an empty string. Sections with empty headers are calledanonymous sections.Example of two anonymous sections:This text actually belongs to an anonymous section,
which happens to also be the first section of this document

[]
This text actually belongs to the second anonymous section
of this document.
If the very first section of a document is anonymous, it can
ignore to define its header.Note:As you noticed, two sections may have similar headers. Read more about this in the 'Section family' section.Base sectionIs called Base section a section as defined in the document, without any transformation, i.e., just text lines. From the source code perspective, the body of such section is represented by default as a list of lines (string).The alternative representation of such body is just a concatenated version of this list.Dict sectionIs called Dict section, a section that is intended to be converted into a dictionary object by the parser.By default, the library uses the regulardicttype to represent the body of such section. This default behavior can be customized through theValueConverterobject.Section familyDifferent sections can have the same header. In this case, they belong to asection family. By default, the.get(header)method from theDocumentclass returns the first defined section with the given header argument. This method actually accepts asub_indexargument which is set to0by default to access the first item of the section family.Get the 2nd section in the 'user' section family:...section=document.get("user",sub_index=1)Get the list of all sections in the 'user' section family:...sections=document.get_all("user")Specs of the Dict SectionThe specs for a Dictionary Section are mostly aligned with Python specs. For example, a Jesth Integer follows exactly the definition of Python Integer.Note:The Jesth library allows the customization of each type of data as well in the context of parsing or in the context of rendering. This customization is made via theValueConverterobjectScalarsJesth supports integers (base 10, hexadecimal, octal, binary), float (approximate binary floating-point, fixed-precision decimal floating-point), complex numbers, and booleans.Note:Jesth Scalars specs are aligned with their respective Python equivalent specs.IntegerJesth Dict section supports Integer whose specs is aligned with Pythoninttype specs.Example:int = -4_200_000
hex_int = 0xAE1F
oct_int = 0o3427
bin_int = 0b101_0011_1010_1010Note:By default, Jesth tidy up numbers by using the underscore convention.Mapping with actual Python objects:JesthPythonintegerinthexadecimal intjesth.box.HexIntsubclassingintoctal intjesth.box.OctIntsubclassingintbinary intjesth.box.BinIntsubclassingintFloatJesth Dict section supports approximate binary floating-point and fixed-precision decimal floating-point which should be represented in the scientific notation.Approximate binary floating-point:float = 3.14fixed-precision decimal floating-point:decimal_float = 3.14E0Note:By default, Jesth tidy up float numbers by using the underscore convention.Decimal float with underscores:decimal_float = 3.141_592_653_589_793_238_46E0Mapping with actual Python objects:JesthPythonbinary floating-pointfloatdecimal floating-pointdecimal.DecimalComplexJesth Dict section supports complex numbers.Example:my_complex = -4.2+6.9iThe number in the example above is equivalent to this in Python:my_complex=-4.2+6.9j# ormy_complex=complex(-4.2,6.9)Note:The only difference between the Jesth complex and Python complex is the usage ofias suffix by Jesth instead ofjlike Python.BoolJesth Dict section supports booleans.Mapping with actual Python objects:JesthPythontrueTruefalseFalseNull valueJesth Dict section supports thenullvalue that translates to PythonNonesingleton.Single-line stringsJesth Dict Section supports single-line strings. Such strings may embed unicode escape sequence and should be enclosed with double-quotes. Raw strings (WYSIWYG) are also supported and in these ones should be enclosed with single-quotes.String embedding unicode escape sequences:my_string = "this is a \n single-line string \u000A with line breaks"
another_string = "string containing "double quotes" and a \\ backslash"Note:The Backslash character is a special one here. It is used to escape a backslash or to make a unicode escape sequence. It shouldn't be used to escape a quote.Raw string (WYSIWYG):my_string = 'this is a \n single-line string \u000A with no line break'
another_string = 'string containing 'single quotes' and a \ backslash'Mapping with actual Python objects:JesthPythonDouble quotes StringstrSingle quotes Stringjesth.box.RawStringThe Jesth single-line String type is aligned with Pythonstrtype.Multi-line stringsMulti-line strings work like single-line strings with one major difference: they can actally span on multilines !Creating multi-line strings implies using tags and indentation. The Jesth name for a multi-line string that supports unicode escape sequences isTextand its tag is(text). For raw multi-line string, the Jesth name isRawand its tag is(raw).A multi-line string is written with a 4-spaces indent under its tag. This indented block under the tag is then closed with a---separator symbol placed on its own line.Multi-line string that supports unicode escape sequences:multiline = (text)
    This is a multi-line
    string that supports \\
    unicode escape sequences \u000A
    ---
    
empty_text = (text)
    ---Raw multi-line string:multiline = (raw)
    This is a raw multi-line \
    string \u000A
    ---
    
empty_raw = (raw)
    ---Mapping with actual Python objects:JesthPythonTextjesth.box.TextsubclassingstrRawjesth.box.RawsubclassingstrDate and timeJesth Dict Section supports date and time (ISO 8601).Date and time:datetime = 2020-10-20T15:35:57Z+01:00
date = 2020-10-20
time = 15:35:57Mapping with actual Python objects:JesthPythondatetimedatetime.datetimedatedatetime.datetimedatetime.timeEmbedded binary dataJesth Dict Section supports binary data. Under the hood, standard base64 (RFC 4648) is used. For convenience, the Jesth parser allows for line breaks and the Jesth renderer outputs base64 lines of 76 characters.Embedding binary data encoded with base64 implies the use of a specific tag and indentation. The Jesth name for binary data isBinand its tag is(bin).Binary data encoded with base64 is written with a 4-spaces indent under the(bin)tag. This indented block under the tag is then closed with a---separator symbol placed on its own line.bin = (bin)
    TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdCwg
    c2VkIGRvIGVpdXNtb2QgdGVtcG9yIGluY2lkaWR1bnQgdXQgbGFib3JlIGV0IGRvbG9yZSBtYWduY
    SBhbGlxdWEu
    ---
    
empty_bin = (bin)
    ---Mapping with actual Python objects:JesthPythonbin (base64)bytesComments and whitespacesJesth Dict Section supports comments which are strings starting with#. Comments and Whitespaces can be preserved while converting a Jesth Dict Section into a Python Dictionary.Example with comments and whitespaces:# this is a comment
key = "value"

# another comment
number = 42Mapping with actual Python objects:JesthPythoncommentjesth.box.Commentsubclassing ``strwhitespacejesth.box.Whitespacesubclassing ``strNote:Along withjesth.box.Commentandjesth.box.Whitespace, the classesjesth.box.CommentIDandjesth.box.WhitespaceIDare also used to help embed comments and whitespace in a Python dictionary by playing the role of keys. They don't accept any argument, subclassint. Each instanciation of one of these classes create a new unique ID. The creation of this unique ID is thread-safe and handled byjesth.box.N.Collections (nested dict and list)Jesth Dict Section supports nested dictionaries and lists.Nesting a collection implies the use of a specific tag and indentation. The Jesth name for a nested dictionary isDictand its tag is(dict). In the case of a nested list, its Jesth name isListand its tag is(list).Nested collections:my_dict = (dict)
    key_1 = "value"
    # Nested list
    key_2 = (list)
        "item i"
        "item ii"
        # nested dict
        (dict)
            project = "jesth"
            number = 42Mapping with actual Python objects:JesthPythondictdictlistlistApplication programming interfaceThe library exposes functions and classes to load and dump Jesth document with optional customization at each stage. Loaded documents and their sections are encapsulated into classes exposing methods and properties with a high emphasis on ergonomy.Loading JesthTo load Jesth data, one may use thejesth.parser.readfunction when the source is a file (file-like object, path string, or pathlib.Path object), orjesth.parser.parsefunction when the source is a string.These functions returns aDocumentobject that is expected to containSectionobjects. Under the hood, these functions use thejesth.parser.Parserclass that consume line by line the contents and stops when[[END]]is encountered.Both functions accept theValueConverterobject for customization.Examples:fromdecimalimportDecimalfromjesthimportread,parse# Using 'read'path="/path/to/jesthfile"document=read(path)# path may also be a pathlib.Path object# Using 'parse'withopen(path,"r")asfile:text=file.read()document=parse(text)# Force floats to use decimal.Decimalvc=ValueConverter(float_decoder=Decimal)document=parse(text,value_converter=vc)As showed in the example above, references to these functions are available from the root module.Dumping JesthTo save the contents of aDocumentin a file, use thejesth.renderer.writefunction. This function uses another functionjesth.renderer.renderwhich takes a document as argument then return its textual representation (with the square brackets and all).Note:TheDocumentclass itself already exposes therenderandsavemethods which use under the hood the functionsrenderandwritethat has been described above.Example:fromjesthimportread,write,renderpath="/path/to/jesthfile"document=read(path)# Save to same filedocument.save()# Save to another filenew_path="/path/to/copy.txt"document.save_to(new_path)# render (convert the document into text)text=document.render()# note that the 'render' method accepts *headers# and spacing arguments.# By default the spacing is 1 (space) between sections# Save to a file with 'write' functionwrite(document,path)# path may be a pathlib.Path instance# render with the 'render' functiontext=render(document)As showed in the example above, references to thewriteandrenderfunctions are available from the root module.The Document objectThereadandparsefunctions returns a document object which contains the list of sections objects and exposes methods to edit the document, save it in a file, render its contents etc.Properties:path,sections,value_converter, andheaders.Edit a documentThe document object exposesappend,insert,remove, andremove_allmethods to add and remove sections.Example:fromjesthimportreadpath="/path/to/jestfile"document=read(path)# append a new section to the documentbody=["line 1","line 2","line 3"]document.append("header",body=body)# body may be a dict or string# insert an empty section at a specific indexdocument.insert(0,"header")# therefore, first section# override the first section with a new empty onedocument.set(0,"header")# 'body' is still set to None here# remove the last section with a specific headerdocument.remove("header")# sub_index defaults to -1# remove the first section with a specific headerdocument.remove("header",sub_index=0)# note that the index argument is meant to be relative# to this 'section family', i.e., sections with same header.# remove all sections by headerdocument.remove_all("header")Query a documentThe document object exposesget,get_all, andcountmethods to retrieve sections or their count.Example:fromjesthimportreadpath="/path/to/jestfile"document=read(path)# get the first section with a specific headersection=document.get("header")# the index arg defaults to 0# get the last section with a specific headersection=document.get("header",sub_index=-1)# note that the index argument is meant to be relative# to this 'section family', i.e., sections with same header.# get all sections with a specific headersections=document.get_all("header")# count the number of sections sharing a specific headern=document.count("header")The Section objectThe Section object contains the header and the body of a section. The body is stored as a list of string that may be converted on demand, if compatible, into a Python dictionary.Properties:header,body, andvalue_converter.Example:fromjesthimportreadpath="/path/to/jestfile"document=read(path)section=document.get("header")# Convert the body of this section into a dict then return it.# Beware, this method won't raise an exception at all# instead, it may return a fallback valuedata=section.to_dict()# accepts "default", "fallback", "strict"# Concatenate the lines of the body to form a text stringdata=section.to_text()# Convert the body of this section into a dict then return it.# Beware, this method will raise an exception if the conversion failsdata=section.create_dict()# accepts "default" and "strict"# the default parameter is for default data if the new created dict# is empty. The strict parameter is to tell whether comment/whitespace# should be preserved and inserted in the new dict !# update the body of this sectionbody=["line 1","line 2","line 3"]section.update(body)# here, body may be a string or a dictThe ValueConverter objectThis object has methods to encode and decode values, and also properties to customize conversion types.Most time, you won't need to set or edit this object. The encode and decode methods of this object are only useful for test purpose.Parameters:dict_type: the Python type in which a Jesth Dict should be converted into. Defaults to Python dict type.list_type: the Python type in which a Jesth List should be converted into. Defaults to Python list type.XXX_types: this represents a group of parameters. Here, XXX is a placeholder for a Jesth data type. Valid types are: dict, list, bin, bool, complex, date, datetime, float, integer, raw, string, text, time. Examples: dict_types, float_types, and time_types. Use this parameter to set a list of Python types that may be encoded in the Jesth type (the same used as prefix). Example: dict_types defaults to [dict, OrderedDict], i.e., while encoding some Python data, an OrderedDict instance or a regular dict instance will be encoded as a Jesth dict.XXX_encoder: this represents a group of parameters. Here, XXX is a placeholder for one of: bin, bool, complex, date, datetime, float, integer, null, raw, string, text, time, fallback. Use these parameters to set a callable to encode Python values into Jesth values. Example: float_encoder = decimal.DecimalXXX_decoder: this represents a group of parameters. Here, XXX is a placeholder for one of: bin, bool, complex, date, datetime, float, integer, null, raw, string, text, time, fallback. Use these parameters to set a callable to decode Jesth values into Python values. Example: integer_decoder = intNote:the parameters of ValueConverter are mirrored as read and write properties.Jesth Schema for data validationComing soon... ;)Available implementationsFor the moment, the only available implementation of Jesth is this reference library written in Python.
If you are interested into writing a Jesth library for another language (Rust, C#, Java, JavaScript, PHP, etc), feel free to get in touch withmefor any question !MiscellaneousCollection of miscellaneous notes.AcronymyJesth stands forJust Extract Sections Then Hack'em. Here, the wordhackis a direct reference to theHacker culture.End of documentTo tell the parser to stop parsing a document, input in the document the[[END]](case-insensitive) tag on its own line.Opening square bracket at the beginning of a lineYou must escape the opening square brackets at the start of a line of a body with a backslash\character. And whenever your line needs to start withnbackslash(es) followed by an opening square bracket, you need to prepend an extra backslash to the line, resulting inn+1backslash(es).Note:This does not apply when programmatically submitting the body of a section as an argument to a function or class.Example:[header]
\[hello world !
\\\[Here there is one extra line to escape the bracket. Backslashes themselves shouldn't be escapedIndentationIndentation in a Jesth file is aligned with the de-facto indentation convention in Python. Four spaces represent one indent.Beautiful cover imageThe beautiful cover image is made with the amazingcarbon app.ContributingFeel free to open an issue to report a bug, suggest some changes, show some useful code snippets, or discuss anything related to this project. You can also directly emailme.Note:I would like to automate a decent Git workflow (branching, merging patches, all the basic cool things cool kids do) withBackstage. In the meantime, I will manually incorporate the fixes into the only existing branch (master). Thank you for your understanding !Setup your development environmentFollowing are instructions to setup your development environment# create and activate a virtual environmentpython-mvenvvenvsourcevenv/bin/activate# clone the project then change into its directorygitclonehttps://github.com/pyrustic/jesth.gitcdjesth# install the package locally (editable mode)pipinstall-e.# run testspython-munittestdiscover-f-stests-t.# deactivate the virtual environmentdeactivateInstallationJesthiscross platform. It is built onUbuntuwithPython 3.8and should work onPython 3.5ornewer.Create and activate a virtual environmentpython-mvenvvenvsourcevenv/bin/activateInstall for the first timepipinstalljesthUpgrade the packagepipinstalljesth--upgrade--upgrade-strategyeagerDeactivate the virtual environmentdeactivateAbout the authorHi, I'm Alex, a tech enthusiast ! Get in touch withme!Back to top