Package: pon
Python Object Notation (PON) fills the need for simple, human usable
and maintainable, configuration data file format for Python.A basic example of a PON file:dict(
  version = [1, 2, 3],  # Major, Minor and Micro version number
  email = "avdn@europython.org",
  score = 8.7,
  restrictions = None,
  published = True,
  install_requires=dict(   # nested structure
    any=['requests', 'beautifulsoup'],
    py26=['ordereddict'],
  ),
  oddkeys = { "2":  2, "s p a c e d": "out"}
)PON:is readable and maintainable through familiarity for all Python programmersallows commentsis editable by humans without breaking things every other timeis efficiently and securely loadableHas several basic types: string, numbers (including simple math on
the numbers),True,False,None,date(),datetime(in UTC, no timezone info),basic structures: dict (dict()/{}) and list ([]),extended structures: set (set()/{}(the latter only on
2.7+) and tuples (( ))basic and extended structures which can be nested (within its own
type as well as in the other types) with the following restrictions:top level must be a dictno syntax for self references{ }can only having strings as keyschildren of the extended structures (set, tuple) cannot be
referenced for substitutionshas substitutions/interpolations on string values, with the
substitute value, being thestr()result of a possible complex valuecan be embedded in a Python file as it is valid Python code, and
provides routines to extract such an embedded configuration.expects UTF-8 based inputhas support for multi-line strings through normal triple-quoting, as
well as fordedent-ing those multi-line stringshas a read-only parser with a small enough footprint that you do not
require a package to be installed before you can use it
(e.g. ~100 line parser in severalsetup.pyfiles in
packages on PyPI)with a few restrictions, supports round-trippingwithout the loss of
commentsandwithout the loss of carefully ordered keysOutput is PEP8 compliant, except for allowing (but not requiring)
spaces around the=following adict()keyWhy not YAML/XML/INI-CFG/JSONYAMLYAML fulfils all of the requirements of the PON except for the string
substitution on values. And it can fulfil more complex tasks, like
handling (self) references and complex dict/mapping keys. However the
basics are easily readable, not all of its syntactical variations
might be familiar.The main problem is that it requires an external library (ruamel.yaml
for YAML1.2 (with round-trip preservation of comments), PyYAML for 1.1
(with guaranteed loss of comments)) and e.g. cannot be used within asetup.py.XMLAn XML parser ships with Python, so using that from any program and fromsetup.pyis possible. XML combines the inefficiency of ASCII with the
illegibility of binary. ‘nuff said.INIINI/CFG files are familiar to a lot of people, but unfortunately there are
all kinds of variations and extensions. Python’s built-inconfigparser(ConfigParserin < Python 3.0), provides only limited structural
information. It is essentially a dict, with the sections as keys for values
that are a dict (the key = value pairs). It supports multi-line strings in a
traditional way (indented continuation lines), comments and has interpolation
in two different forms.JSONJSON is a format familiar to many for data exchange. However several
things make it unusableq for configuration files that are manipulated by
humans:the inability to handle comments in most implementations
(including Python’sjsonmodule), although the inventor specified
these should be ignoredhumans generating an invalid file on every other edit, because
commas are separators, and are not allowed before list/dict terminal]resp.}JSON requires even simple dict keys (strings without spaces) to be
obfuscated by quotesPython has most of the batteries for PON includedIn Python you can already do:config = dict(
    version = [1, 2, 3],  # Major, Minor and Micro version number
    email = "avdn@europython.org",
    score = 8.7,
    restrictions = None,
    published = True,
    install_requires=dict(   # nested structure
        any=['requests', 'beautifulsoup',],
        py26=['ordereddict']
    ),
)Thisisvalid Python code, and the embedding requirement is
implicitly taken care of.  The problem is you don’t always want to
evaluation/include/import thisas code, because of security
implications. You need to be able to parse this and reject invalid or
dangerous constructs.(Also note that you can delete the whole line containingpy26=in the above without breaking things, although you end up with a comma
before the closing parenthesis.)Parsing out PON (almost) of the boxThe functionliteral_evalfrom theastmodule can parse a more
JSON like variation of the previous config. E.g. the contents of the
following file:{
  "version": [1, 2, 3],  # Major, Minor and Micro version number
  "email": "avdn@europython.org",
  "score": 8.7,
  "restrictions": None,
  "published": True,
  "install_requires": {   # nested structure
    "any": ['requests', 'beautifulsoup',],
    "py26": ['ordereddict']
  },
}using the following:python -c 'import ast; ast.literal_eval(open("input2.pon").read())The above can also be relatively easily parsed from a larger (Python
source) file by looking for the assignment to a known variable,config = {and the corresponding}(usually at the same indentation level).This is almost JSON, but to be able to include JSON in Python, as well as
being able to parse that, you need to change it some more:true = True
null = None
config = {
    "version": [1, 2, 3],
    "email": "avdn@europython.org",
    "score": 8.7,
    "restrictions": null,
    "published": true,
    "install_requires": {
        "any": ["requests", "beautifulsoup",],
        "py26": ["ordereddict"]
    }
}You have to definenullandtruefor the Python parser to accept
this. For most JSON parsers you also have to remove the comments, and
consistently use double quotes for strings. And above all you have to remove
trailing commas, which is most often forgotten when deleting whole key-value
lines at the end of a dictionary/mapping in JSON (resulting in non-running
programs unless you use ruamel.yaml/PyYAML to load your JSON files).Actually, the above is not valid JSON (did you see the trailing comma
in the list on theanyline?). These problems don’t make JSON a
bad format. It is fine for information interchange between
programs. JSON files should just never be edited, and preferably not
even have to be read, by humans.A replacement for literal_evalast.literal_evalcannot deal withdict(), so using that you
cannot have keys that are strings without quotes. It
also throws a useless generic ValueError, when it is fed invalid
strings, making it difficult to provide meaningful feedback to the
human editor of the (invalid) configuration data. And finally it
happily tries and fails to do its job when you feed it nonsense data
like a float.ast.literal_evalis a good example how you can make a minimal
evaluator around theastfacilities. A small adaptation can handle
the extras likedict,dateanddatetime. Thus allowing
non-quoted simple keys, while disallowing non string keys for{},
forcing a toplevel dictionary. The code, including adaptations for 2.6
and later support (2.6’s cannot handle{}type sets):import sys                                               # NOQA
import platform                                          # NOQA
import datetime                                          # NOQA
from textwrap import dedent                              # NOQA
from _ast import *                                       # NOQA

if sys.version_info < (3, ):
    string_type = basestring
else:
    string_type = str

if sys.version_info < (3, 4):
    class Bytes():
        pass

    class NameConstant:
        pass

if sys.version_info < (2, 7) or platform.python_implementation() == 'Jython':
    class Set():
        pass


def loads(node_or_string, dict_typ=dict, return_ast=False, file_name=None):
    """
    Safely evaluate an expression node or a string containing a Python
    expression.  The string or node provided may only consist of the following
    Python literal structures: strings, bytes, numbers, tuples, lists, dicts,
    sets, booleans, and None.
    """
    if sys.version_info < (3, 4):
        _safe_names = {'None': None, 'True': True, 'False': False}
    if isinstance(node_or_string, string_type):
        node_or_string = compile(
            node_or_string,
            '<string>' if file_name is None else file_name, 'eval', PyCF_ONLY_AST)
    if isinstance(node_or_string, Expression):
        node_or_string = node_or_string.body
    else:
        raise TypeError("only string or AST nodes supported")

    def _convert(node, expect_string=False):
        if isinstance(node, (Str, Bytes)):
            return node.s
        if expect_string:
            pass
        elif isinstance(node, Num):
            return node.n
        elif isinstance(node, Tuple):
            return tuple(map(_convert, node.elts))
        elif isinstance(node, List):
            return list(map(_convert, node.elts))
        elif isinstance(node, Set):
            return set(map(_convert, node.elts))
        elif isinstance(node, Dict):
            return dict_typ((_convert(k, expect_string=True), _convert(v)) for k, v
                            in zip(node.keys, node.values))
        elif isinstance(node, NameConstant):
            return node.value
        elif sys.version_info < (3, 4) and isinstance(node, Name):
            if node.id in _safe_names:
                return _safe_names[node.id]
        elif isinstance(node, UnaryOp) and \
             isinstance(node.op, (UAdd, USub)) and \
             isinstance(node.operand, (Num, UnaryOp, BinOp)):  # NOQA
            operand = _convert(node.operand)
            if isinstance(node.op, UAdd):
                return + operand
            else:
                return - operand
        elif isinstance(node, BinOp) and \
             isinstance(node.op, (Add, Sub, Mult)) and \
             isinstance(node.right, (Num, UnaryOp, BinOp)) and \
             isinstance(node.left, (Num, UnaryOp, BinOp)):  # NOQA
            left = _convert(node.left)
            right = _convert(node.right)
            if isinstance(node.op, Add):
                return left + right
            elif isinstance(node.op, Mult):
                return left * right
            else:
                return left - right
        elif isinstance(node, Call):
            func_id = getattr(node.func, 'id', None)
            if func_id == 'dict':
                return dict_typ((k.arg, _convert(k.value)) for k in node.keywords)
            elif func_id == 'set':
                return set(_convert(node.args[0]))
            elif func_id == 'date':
                return datetime.date(*[_convert(k) for k in node.args])
            elif func_id == 'datetime':
                return datetime.datetime(*[_convert(k) for k in node.args])
            elif func_id == 'dedent':
                return dedent(*[_convert(k) for k in node.args])
        elif isinstance(node, Name):
            return node.s
        err = SyntaxError('malformed node or string: ' + repr(node))
        err.filename = '<string>'
        err.lineno = node.lineno
        err.offset = node.col_offset
        err.text = repr(node)
        err.node = node
        raise err
    res = _convert(node_or_string)
    if not isinstance(res, dict_typ):
        raise SyntaxError("Top level must be dict not " + repr(type(res)))
    if return_ast:
        return res, node_or_string
    return resThe above 109 lines of Pythonisthe actual code, that loads the full PON from an iterable.This code can be further reduced if you only need to support later
Python versions, and if you know your input is restricted (no math, no
set/tuples/{}, no datetime, etc)SyntaxErrorTheast.literal_evalgives you a generic ValueError without any
indication of what might be wrong nor where things are wrong. From theSyntaxErrorthat is raised on erroreous input byloads()you
can retrieve useful line information:error_str = u"""
dict(
    a= u"α",
    b= False,
    c= date(2015, 9, 12),
    d= 1.37,
)
"""

try:
    loads(error_str)
except SyntaxError as e:
    context = 2
    from_line = e.lineno - (context + 1)
    to_line = e.lineno + (context - 1)
    w = len(str(to_line))
    for index, line in enumerate(error_str.splitlines(True)):
        if from_line <= index <= to_line:
            print(u"{:{}}: {}".format(index, w, line), end=u'')
            if index == e.lineno - 1:
                print(u"{:{}}  {}^--- {}".format(
                    u' ', w, u' ' * e.offset, e.node))giving you:2:    a= u"α",
3:    b= False,
4:    c= date(2015, 9, 12),
         ^--- <_ast.Call object at 0x7f1598d20950>
5:    d= 1.37,
6: )(the PON parser as indicated above extendsast.literal_evalwithdate()and doesn’t throw an eror on that input)MotivationThe development of theliteral_evalextension/replacement was
motivated by cleaning providing version and other information from the__init__.pyof a package to itssetup.pyfile, thereby
minimising the clutter of extra configuration files in the base
directory (it is bad enough withsetup.py,distandtox.inias non hidden files/directories.A version number can be easily parsed from an__init__.pyfile.
But allowing for more complex and complete configuration data allowssetup.pyto be the same for all of my projects.Using theponpackageTheponpackage provides the the main parserloads(), the utility functionsget(),store()andextract()and the PON class (for which the utilities are shortcuts).get()andstore()If you have a configuration:dict(
    a = dict(
        b = 24,
        c = [1, 3.14, {'d': 'klm'}],
}loaded into a variableconfig, you can access the valueklmin
the normal Python way by usingconfig['a']['c'][2]['d']. PON also
provides the functionget()with which you can access the same
value usingget(config, 'a.c.2.d').Based on the nested structure ofconfigthe “2” in that sequence is
converted to an index. As indicated before, integers as dict keys, are not allowed,
keys have to strings.Complementary there is thestore()function (set()being a reserved word in Python)
that takes as third parameter a value, to set or overwrite an existing one:store(config, 'a.c.2.d', 'xyz')Substitution withget()Substitution (called interpolation inConfigParser/configparser) is done by accessing a value of
your configuration with withget(), and providing the extra
keywordexpand. Substitution is done recursively on the expanded value.
You can provide theconfigobject itself to expand:val = get(config, 'some.path', expand=config)and since this is such a common use case, you can specifyexpand=Trueinstead of actually passing inconfigtwice.The syntax for substitution is the usual Python,"{key}".format(key=value), string formatting but the key can be a
dotted sequence valid forget():import pon

config = pon.loads("""\
dict(
    a = dict(
        image = "http://{domain}/images",
        alt = "europython.eu",
        dd = (2011, 10, 2)  # this is a tuple
    ),
    domain = 'python.{tld.organisations}',
    datestr = 'date{a.dd}',
    tld = {"organisations": "org", "commmercian": "com"}
)
""")

for key in ['a.image', 'datestr']:
    print(key, '->', pon.get(config, key, expand=True))giving:a.image -> http://python.org/images
datestr -> date(2011, 10, 2)The recursion for this is restricted to 10 levels.The separator (by default “.”) can be set on thePONclass. Since “:”
is special in format strings, that character cannot be used as separator.RoundTripping PONWith some restrictions it is possible to round trip PON, while
preserving comments, in the smae way ruamel.yaml can for YAML:you will not lose any dataon the first round-trip your formatting might changea second round-trip will result in the same output as the first round-tripIn order to facilitate round-tripping extra information needs to be
kept that is not available in the normal dict you get from the loading
of your PON data structure into Python. This extra processing can be
done up-front, after which the original configuration data is no
longer necessary in text form, but wastes time during loading in case
the round-tripping is never needed. It can be extracted on demand, but
in that case the original textual data needs to be available. This
time vs storage trade of is currently done at load time, and only when
using the PON class (and not when using utility functionloads()).If you create a PON object frominput(a file, a string or a list
of strings) usingPON(input)the resulting object will have
information about the dict keys and list elements and has comment
information associated with these keys.The primary purpose for round-tripping is updating existing
information in the configuration: updating one of the tuple values for
key “version”, adding a dependency package to the list necessary forpy26. If a whole new configuration file, including comments, needs
to be generated, this can generally be done more easily by using (or
starting from) a text template than to try and procedurally built the structure.CommentsDumping the loaded PON structure as text, assuming some formatting
criteria, is relatively easy, if we could just ignore the fact that
comments are important for future readers of the dumped information.The Python built-incompile()function generating the AST
information from which the object holding the configuration
information is extracted, throws away the comments. So the comment
information has to be re-associated with the object, and in addition to
determining what comment belongs where, this requires that the
elements in the object tree can be extended (requiring more complex
objects that behave like dicts/lists, but have extra slots for comment
information, a method which is also used in ruamel.yaml), or that a shadow
structure is kept in the same form as the configuration object.PON follows a hybrid by requiring the dictionaries to have key
insertion ordering (the ordering of the keys in the source
configuration data) as well as keeping a shadow structure. The shadow
structure is extracted from the AST tree (used for generating/checking
the configuration loaded) with tokenization information (which
provides the comments).Comments are associated with dictionary keys or list elements as far
as these are “on their own line”. A full line comment belongs, or
consecutive comments belong, to the next key/element if it is on a
line of its own after the previous key/element. An end-of-line comment
follows a key/element at the end of a line (and there can be only
one). Additionally track is kept of comments before the initial, top
level,dict(, after the final key (there is no following one to
hook it up to) and after the dict closing)token.An example of a heavily commented PON file:# this is the configuration driving setup
# initials comments going before the configuration information
dict(   # the top level dict can also have an end-of-line comment
    # full comment associated with the key version
    # this doesn't explain its usage that much, also associates with version
    version=(1, 2, 3)    # end-of-line comment for version
    alt = dict(   # this is end-of-line for alt
        # associated with place
        place=u"Düsseldorf",
        taste="awful",
        # the next key is klm, this comment  will move down on round-trip.
    ),  # this is assocated with klm as well, even though not a end-of-line
    "klm" = [ 3, 5, 6 ],  # although list elements, belongs to klm
    "xyz" = [   # belong to xyz
        42,     # the answer (associated with 42)
        196,
    ],
    # trailing comment, special
)
# still more to say, special as wellIf you change dictionary keys, comments associated with these will
generally get lost. So do comments associated with key/element that
get deleted.Changes on first round tripPartly due to thepprintcode on which the dumper is based, partly
due to arbitrary decisions on what kind of formatting info is
preserved and partly depending on your input, the following happens on
the first round-trip:the last element on a multi-line dict/list gets a trailing commacomments that cannot be associated with an dict key or list element
on the same line get moved to the next key/element (or the end of
the main dict if no following elements)strings are single quoted unless they contain a single quote (and no
double quotes)indent levels are a at 4 spacesspace around the equal sign between dict keys and values is removedsets and tuples elements cannot be associated with comments, hence comments
wander if they are not on the same line as a dict-key/list-elementsets and tuples are dumped on a single line, any dicts and lines
underneath them are currently inaccessible forget()and
therefore keys/elements for such dicts/lists cannot be associated
with commentsextra lines with white space are silently droppedadd/subtract/multiply is not preservedthe datetimerepr()drops trailing milliseconds if and seconds if
they equal zero.No data or comments get lost, unless you manipulate dict keys and/or
list length. And if the output from a dump is taken as source there
should be no further “wandering”. The following input:try:
    from cStringIO import StringIO as _StringIO
except ImportError:
    from io import StringIO as _StringIO

from pon import PON

input = """
dict(
    pckgs = dict(
        any=['package1', 'package2'],
        py26=['another package', 'and one with a long name',
        'and on a new line']   # where do you go?
    ),
)
"""

out1 = _StringIO()
p1 = PON(input)
p1.dump(out1)
print(out1.getvalue())

out2 = _StringIO()
p2 = PON(out1.getvalue())
p2.dump(out2)


print('roundtrip 1: {0}, roundtrip 2: {1}'.format(
    input == out1.getvalue(),
    out1.getvalue() == out2.getvalue()))gives the following output:dict(
    pckgs=dict(
        any=['package1', 'package2'],
        py26=['another package', 'and one with a long name',
            'and on a new line',   # where do you go?
        ],
    ),
)

roundtrip 1: False, roundtrip 2: TrueFurther improvementsThe set and tuple elements could be indexed, and then comments could
be associated with their elements, and multi-line dumping would be
better preserved.Thedump()could take parameters about indentation depth and on
string quoting information.Thedump()output should be PEP8 compliant in principle. But IMO
the removal of spaces around the=in a multi-line keyword
argument assignment fordict()doesn’t make thing more
readable. A parameter to select one or the other would be useful:dict(
   a='1234324',
   b=['xyz', 'klm'],
)is less easy to read than:dict(
   a = '1234324',
   b = ['xyz', 'klm'],
)Keeping the#of comments on multiple consecutive lines aligned,
even if a value was changed before dumping and has become longer.ShowcaseThe following program contains most (if not all) of the facilities
and round-trips:from io import StringIO as _StringIO
from pon import PON

configs = u'''\
# example config
# should contain all types and facilities
dict(
    s='abc',  # single line string
    # multiline string
    mls="""one
    two
    three""",
    mls_dedent=dedent("""
        abc
        def
    """),
    ghi={'A': 1, 'B': 2},
    klm=['Airbus 370', 'Fokker 100'],
    opq=set([2, 3, 5, 7, 9]),
    rst=(0, 1, 1, 2, 3, 5, 8, 13),   # Fibonacci
    m={u'π': 3.14},
    anniversary=date(2011, 10, 2),
    dts=datetime(1919, 12, 1, 13, 45, 4),
    milisec=datetime(1922, 10, 19, 17, 55, 23, 321),
    six=2 + 4,
    secs_per_day=24 * 60 * 60,
    two=-2 - -4,
    # if you want to extend, do it here
)  # and it's over
'''

out = _StringIO()
p = PON(configs)
p.dump(out)
conf_adjust_for_calc = configs
# calculations are not preserved, they don't round trip, so adjust here
for x, y in (('2 + 4', '6'),
             ('24 * 60 * 60', "{}".format(24 * 60 * 60)),
             ('-2 - -4', '2')):
    conf_adjust_for_calc = conf_adjust_for_calc.replace(x, y)
outl = out.getvalue().splitlines(True)
orgl = conf_adjust_for_calc.splitlines(True)
if outl == orgl:
   print('roundtrip 1: equal')
else:
    import difflib
    diff = difflib.unified_diff(orgl, outl, 'input', 'output')
    for line in diff:
        print(line, end='')with output:--- input
+++ output
@@ -3,9 +3,7 @@
 dict(
     s='abc',  # single line string
     # multiline string
-    mls="""one
-    two
-    three""",
+    mls='one\n    two\n    three',
     mls_dedent=dedent("""
         abc
         def