Package: json-syntax
json-syntaxA Python library to translate between JSON compatible structures and native Python
classes using customizable rules.Use caseIf you're like the authors, you tried writing a encoding function that attempted to
encode and decode by interrogating the types at runtime, maybe calling some method likeasdict. This works fine for generating JSON, but it gets sketchy1when trying to decode the same JSON.Further, we have annotations in Python 3! Even if you're not using a type checker, just
labeling the types of fields makes complex data structures far more comprehensible.This library is aimed at projects that have a complex JSON schema that they're trying to
structure using libraries likeattrs.It exploitsgradual typingvia annotations,typinganddataclassesIt expects classes to bestaticallydescribed using typesBut a fallback can be provided to handle data described at runtimeIt provides hooks to normalize legacy inputsIt makes it trivial to extend the library with your own rulesActions and Rules are simply functionsEncoders and decoders can be pickledThe library has no dependencies of its own on python 3.7+It does not actually read or write JSONSupported typesAtoms includingNone,bool,int,float,str.Floats may optionally be represented as strings.Thedecimal.Decimalclass, represented as itself or in string form.Thedatetime.dateanddatetime.datetimeclasses, represented in ISO8601 form.Preliminary support fordatetime.timedeltaas ISO8601 time durations.Subclasses ofenum.Enum, represented by the string names.Also, afaux_enumsrule will accept an Enum type if you just use strings in your
code.Thetyping.Optional[E]type allows a JSONnullto be substituted for a value.Collections includingtyping.List[E],typing.Tuple[E, ...],typing.Set[E]andtyping.FrozenSet[E].The...isliteraland indicates a homogenous tuple, essentially a
frozen list.Thetyping.Dict[K, V]type allows a JSON object to represent a homogenousdict.Restriction: the keys must be strings, ints, enums or dates.New: Thetyping.TypedDicttype allows a JSON object to represent adictwith specific
keys.Python classes implemented usingattrs.attrs,dataclasses.dataclassare
represented as JSON dicts andNamed tuples viatyping.NamedTupleand heterogenous tuples viatyping.Tuple.Though, you should consider converting these todataclass.Thetyping.Union[A, B, C]rule will recognize alternate types by inspection.In addition,dataclassandattrsclasses support hooks to let you completely customize
their JSON representation.ExtrasThese were originally intended as examples for how to use the package, but they're potentially
useful in their own right.A rulesetfor use with AWS DynamoDB is included with basic facilities.Restriction: No general support fortyping.Union, onlyOptional.Restriction: No general support forSet, only the special cases that are native to DynamoDB.AFlagpsuedo-typeallows you to use regular strings directly as flags.A rulethat will accept a completedatetimeand return adateby truncating the timestamp.UsageThis example is also implemented in unit tests. First, let's declare some classes.importjson_syntaxassynfromdataclassesimportdataclass# attrs works toofromdecimalimportDecimalfromdatetimeimportdatefromenumimportEnum@dataclassclassAccount:user:strtransactions:List['Trans']# Forward references work!balance:Decimal=Decimal()classTransType(Enum):withdraw=0deposit=1@dataclassclassTrans:type:TransTypeamount:Decimalstamp:dateWe'll next set up a RuleSet and use it to construct an encoder. Thestd_rulesetfunction is a one-liner with some reasonable overrides. Here, we've decided that because
some intermediate services don't reliably retain decimal values, we're going to
represent them in JSON as strings.>>>rules=syn.std_ruleset(decimals=syn.decimals_as_str)>>>encode_account=rules.python_to_json(typ=Account)>>>decode_account=rules.json_to_python(typ=Account)The RuleSet examines the type and verb, searches its list of Rules, and then uses the
first one that handles that type and verb to produce an Action.For example,attrs_classesis a Rule that recognizes the verbspython_to_jsonandjson_to_pythonand will accept any class decorated with@attr.sor@dataclass.It will scan the fields and ask the RuleSet how to encode them. So when it seesAccount.user, theatomsrule will match and report that converting astrto JSON
can be accomplished by simply callingstron it. The action it returns will literally
be thestrbuiltin.Thusattrs_classeswill build a list of attributes onAccountand actions to convert
them, and constructs an action to represent them.>>>sample_value=Account(...'bob',[...Trans(TransType.withdraw,Decimal('523.33'),date(2019,4,4))...],Decimal('77.00')...)>>>encode_account(sample_value){'user':'bob','transactions':[{'type':'withdraw','amount':'523.33','stamp':'2019-04-04'}],'balance':'77.00'}Encoding and decodingThe aim of all this is to enable reliable usage with your preferred JSON library:withopen('myfile.json','r')asfh:my_account=decode_account(json.load(fh))withopen('myfile.json','w')asfh:json.dump(encode_account(my_account))Using generic typesGenerally, thetypingmodule simple provides capital letter type names that obviously
correspond to the internal types.See TYPES for a more thorough introduction.And you specify the type of the contents as a parameter in square brackets.Thus we have:listandList[E]setandSet[E]tupleandTuple[E, ...]is a special case!frozensetandFrozenSet[E]dictandDict[K, V]Tuple is a special case. In Python, they're often used to mean "frozenlist", soTuple[E, ...](the...isthe Ellipsis object) indicates all elements have
the typeE.They're also used to represent an unnamed record. In this case, you can useTuple[A, B, C, D]or however many types. It's generally better to use adataclass.The standard rules don't support:Using abstract types likeIterableorMapping.Using type variables.Any kind of callable, coroutine, file handle, etc.Support for deriving from GenericThere is experimental support for deriving fromtyping.Generic. Anattrsordataclassmay declare itself a generic class. If another class invokes it asYourGeneric[Param, Param], thoseParamtypes will be substituted into the fields during encoding. This is
useful to construct parameterized container types. Example:@attr.s(auto_attribs=True)
class Wrapper(Generic[T, M]):
    body: T
    count: int
    messages: List[M]

@attr.s(auto_attribs=True)
class Message:
    first: Wrapper[str, str]
    second: Wrapper[Dict[str, str], int]UnionsA union type lets you present alternate types that the converters will attempt in
sequence, e.g.typing.Union[MyType, int, MyEnum].This is implemented in theunionsrule as a so-called2undiscriminated union. It means the module won't add any additional information to the
value such as some kind of explicit tag.When converting from Python to JSON, the checks are generally just usingisinstance,
but when converting from JSON to Python, the check may be examining strings anddictfields.Thus, ambiguous values, especially JSON representations, may confuse the decoder.
See the section onsharp edgesfor more details.HooksWe'll first examine decode and encode hooks. These let us entirely rewrite the JSON
representation before the normal logic is applied.Let's suppose ourAccountclass used to named thebalancefieldbaland we need to
support legacy users.@dataclassclassAccount:@classmethoddef__json_pre_decode__(cls,value):if'bal'invalue:value=dict(value)value['balance']=value.pop('bal')returnvaluedef__json_post_encode__(self,value):returndict(value,bal=value['balance'])...When we decode the value, the following sequence of steps takes place:__json_pre_decode__is called with{'user': 'bob', 'bal': '77.0', ...}and it
returns{'user': 'bob', 'balance': '77.0', ...}Decoders are called againstuserandbalanceand the other fieldsThe resulting dictionary is passed toAccount(**result)to construct the instance.During encoding, the reverse sequence takes place:The instance's fields are read and passed to encoders.The values are combined into adict.__json_post_encode__is called with{'user': 'bob', 'balance': '77.0', ...}and
can adjust the field name tobal.JSON type check hookType checks are only used injson-syntaxto supporttyping.Union; in a nutshell, theunionsrule will inspect some JSON to see which variant is present.If a type-check hook is not defined,__json_pre_decode__will be called before the
standard check is completed. (The standard check attempts to determine if required
fields are present and have the correct type.)If you have information that can determine the type faster, a check hook can help.Going back to our Account example, suppose we decide to support multiple account types
through a specialclassfield. This is faster and more robust.classAbstractAccount:@classmethoddef__json_check__(cls,value):returnisinstance(value,dict)andvalue.get('class')==cls.__name__@dataclassclassAccountA(AbstractAccount):...encode_account=rules.lookup(typ=Union[AccountA,AccountB,AccountC],verb='python_to_json')Adding custom rulesSeethe extrasfor details on custom rules, but generally a rule is just a
function. Say, for instance, your type has class methods that encode and decode, this
would be sufficient for many cases:defmy_rule(verb,typ,ctx):ifissubclass(typ,MyType):ifverb=='json_to_python':returntyp.decoderelifverb=='python_to_json':returntyp.encoderIf your rule needs an encoder or decoder for a standard type, it can callctx.lookup(verb=verb, typ=subtype). The helper functions defined injson_syntax.action_v1are intended to stay the same so that custom rules can reuse them.Debugging amibguous structures(May need more docs and some test cases.)Asjson-syntaxtries to directly translate your Python types to JSON, it is possible
to write ambiguous structures. To avoid this, there is a handyis_ambiguousmethod:# This is true because both are represented as an array of numbers in JSON.rules.is_ambiguous(typ=Union[List[int],Set[int]])@dataclassclassAccount:user:straddress:str# This is true because such a dictionary would always match the contents of the account.rules.is_ambiguous(typ=Union[Dict[str,str],Account])The aim of this is to let you put a check in your unit tests to make sure data can be
reliably expressed given your particular case.Internally, this is using thePATTERNverb to represent the JSON pattern, so this may
be helpful in understanding howjson-syntaxis trying to represent your data:print(rules.lookup(typ=MyAmbiguousClass,verb='show_pattern'))Sharp edgesThe RuleSet caches encoders.Construct a new ruleset if you want to change settings.Encoders and decoders do very little checking.Especially, if you're translating
Python to JSON, it's assumed that your Python classes are correct. The encoders and
decoders may mask subtle issues as they are calling constructors likestrandintfor you. And, by design, if you're translating from JSON to Python, it's assumed you
want to be tolerant of extra data.Everything to do with typing.It's a bit magical and sort of wasn't designed for this.We have a guide to it to try and help.Union types.You can usetyping.Unionto allow a member to be one of some number of
alternates, but there are some caveats. You should use the.is_ambiguous()method of
RuleSet to warn you of these.Atom rules accept specific types.At present, the rules for atomic types (int,str,bool,date,float,Decimal) must be declared as exactly those types. With
multiple inheritance, it's not clear which rule should applyChecks are stricter than converters.For example, a check forintwill check whether
the value is an integer, whereas the converter simply callsinton it. Thus there are
inputs for whereMyTypewould pass butUnion[MyType, Dummy]will fail. (Note
thatOptionalis special cased to look forNoneand doesn't have this problem.)Numbers are hard.See the rules namedfloats,floats_nan_str,decimals,decimals_as_strfor details on how to get numbers to transmit reliably. There is no rule for
fractions or complex numbers as there's no canonical way to transmit them via JSON.MaintenanceThis package is maintained via thepoetrytool. Some useful commands:Setup:poetry installRun tests:poetry run pytest tests/Reformat:black json_syntax/ tests/Generate setup.py:dephell deps convert -e setupGenerate requirements.txt:dephell deps convert -e reqRunning tests via dockerThe environments for 3.4 through 3.9 are inpyproject.toml, so just run:dephell deps convert -e req  # Create requirements.txt
dephell docker run -e test34 pip install -r requirements.txt
dephell docker run -e test34 pytest tests/
dephell docker shell -e test34 pytest tests/
dephell docker destroy -e test34Notes1: Writing the encoder is deceptively easy because the instances in
Python have complete information. The standardjsonmodule provides a hook to let
you encode an object, and another hook to recognizedicts that have some special
attribute. This can work quite well, but you'll have to encodeallnon-JSON types
with dict-wrappers for the process to work in reverse.↩2: A discriminated union has a tag that identifies the variant, such as
status codes that indicate success and a payload, or some error. Strictly, all unions
must be discriminated in some way if different code paths are executed. In theunionsrule, the discriminant is the class information in Python, and the structure of the JSON
data. A less flattering description would be that this is a "poorly" discriminated
union.↩