Package: vtjson
vtjsonA lightweight package for validating JSON like Python objects.SchemasValidation of JSON like Python objects is done according to aschemawhich is somewhat inspired by a typescript type. The format of a schema is more or less self explanatory as the following example shows.ExampleBelow is the schema of the run object in the mongodb database underlying the Fishtest web applicationhttps://tests.stockfishchess.org/testsimportmathfromdatetimeimportdatetimefrombson.objectidimportObjectIdfromvtjsonimportip_address,number,regex,urlnet_name=regex("nn-[a-z0-9]{12}.nnue",name="net_name")tc=regex(r"([1-9]\d*/)?\d+(\.\d+)?(\+\d+(\.\d+)?)?",name="tc")str_int=regex(r"[1-9]\d*",name="str_int")sha=regex(r"[a-f0-9]{40}",name="sha")country_code=regex(r"[A-Z][A-Z]",name="country_code")run_id=regex(r"[a-f0-9]{24}",name="run_id")worker_info_schema={"uname":str,"architecture":[str,str],"concurrency":int,"max_memory":int,"min_threads":int,"username":str,"version":int,"python_version":[int,int,int],"gcc_version":[int,int,int],"compiler":{"clang++","g++"},"unique_key":str,"modified":bool,"ARCH":str,"nps":number,"near_github_api_limit":bool,"remote_addr":ip_address,"country_code":{country_code,"?"},}results_schema={"wins":int,"losses":int,"draws":int,"crashes":int,"time_losses":int,"pentanomial":[int,int,int,int,int],}schema={"_id?":ObjectId,"start_time":datetime,"last_updated":datetime,"tc_base":number,"base_same_as_master":bool,"results_stale?":bool,"rescheduled_from?":run_id,"approved":bool,"approver":str,"finished":bool,"deleted":bool,"failed":bool,"is_green":bool,"is_yellow":bool,"workers?":int,"cores?":int,"results":results_schema,"results_info?":{"style":str,"info":[str,...],},"args":{"base_tag":str,"new_tag":str,"base_net":net_name,"new_net":net_name,"num_games":int,"tc":tc,"new_tc":tc,"book":str,"book_depth":str_int,"threads":int,"resolved_base":sha,"resolved_new":sha,"msg_base":str,"msg_new":str,"base_options":str,"new_options":str,"info":str,"base_signature":str_int,"new_signature":str_int,"username":str,"tests_repo":url,"auto_purge":bool,"throughput":number,"itp":number,"priority":number,"adjudication":bool,"sprt?":{"alpha":0.05,"beta":0.05,"elo0":number,"elo1":number,"elo_model":"normalized","state":{"","accepted","rejected"},"llr":number,"batch_size":int,"lower_bound":-math.log(19),"upper_bound":math.log(19),"lost_samples?":int,"illegal_update?":int,"overshoot?":{"last_update":int,"skipped_updates":int,"ref0":number,"m0":number,"sq0":number,"ref1":number,"m1":number,"sq1":number,},},"spsa?":{"A":number,"alpha":number,"gamma":number,"raw_params":str,"iter":int,"num_iter":int,"params":[{"name":str,"start":number,"min":number,"max":number,"c_end":number,"r_end":number,"c":number,"a_end":number,"a":number,"theta":number,},...,],"param_history?":[[{"theta":number,"R":number,"c":number},...],...,],},},"tasks":[{"num_games":int,"active":bool,"last_updated":datetime,"start":int,"residual?":number,"residual_color?":str,"bad?":True,"stats":results_schema,"worker_info":worker_info_schema,},...,],"bad_tasks?":[{"num_games":int,"active":False,"last_updated":datetime,"start":int,"residual":number,"residual_color":str,"bad":True,"task_id":int,"stats":results_schema,"worker_info":worker_info_schema,},...,],}ConventionsAs in typescript, a (string) key ending in?represents an optional key. The corresponding schema (the item the key points to) will only be used for validation when the key is present in the object that should be validated. A key can also be made optional by wrapping it asoptional_key(key).If in a list/tuple the last entry is...(ellipsis) it means that the next to last entry will be repeated zero or more times. In this way generic types can be created. For example the schema[str, ...]represents a list of strings.UsageTo validate an object against a schema one can simply dovalidate(schema,object)If the validation fails this will throw aValidationErrorand the exception contains an explanation about what went wrong. The full signature ofvalidateisvalidate(schema,object,name="object",strict=True)The optionalnameargument is used to refer to the object being validated in the returned message.The optional argumentstrictindicates whether or not the object being validated is allowed to have keys/entries which are not in the schema.WrappersA wrapper takes one or more schemas as arguments and produces a new schema.Warning:wrappers should be considered as being immutable. In other words: if the underlying schemas change after a wrapper has been created then the effect of validating against the wrapper is undefined. If youreallyneed this then you should recreate the wrapper.An object matches the schemaunion(schema1, ..., schemaN)if it matches one of the schemasschema1, ..., schemaN. This is almost the same as{schema1, ..., schemaN}, or equivalentlyset((schema1, ..., schemaN))ifschema1, ..., schemaNare hashable.An object matches the schemaintersect(schema1, ..., schemaN)if it matches all the schemasschema1, ..., schemaN.An object matches the schemacomplement(schema)if it does not matchschema.An object matches the schemalax(schema)if it matchesschemawhen validated withstrict=False.An object matches the schemastrict(schema)if it matchesschemawhen validated withstrict=True.An object matches the schemaset_name(schema, name)if it matchesschema. But thenameargument will be used in non-validation messages.An object matches the schemacompile(schema)if it matchesschema.vtjsoncompiles the schema before performing a validation so pre-compiling is not necessary but, in some cases, it may gain a bit of performance.An object matches the schemaquote(schema)if it is equal toschema. For example the schema{"cats", "dogs"}matches the strings"cats"and"dogs"but the schemaquote({"cats", "dogs"})matches the set{"cats", "dogs"}.Built-insSome built-ins take arguments. If no arguments are given then the parentheses can be omitted. Soemailis equivalent toemail().regex(pattern, name=None, fullmatch=True, flags=0). This matches the strings which match the given pattern. The optionalnameargument may be used to give the regular expression a descriptive name. By default the entire string is matched, but this can be overruled via thefullmatchargument. Theflagsargument has the usual meaning.interval(lowerbound, upperbound). This checks iflowerbound <= object <= upperbound, provided the comparisons make sense. An upper/lowerbound...(ellipsis) means that the
corresponding inequality is not checked.number. Matchesintandfloat.email. Checks if the object is a valid email address. This uses the packageemail_validator. Theemailschema accepts the same options asvalidate_emailin loc. cit.ip_addressandurl. These are similar toemail.domain_name(ascii_only=True, resolve=False). Checks if the object is a valid domain name. Ifascii_only=Falsethen allow IDNA domain names. Ifresolve=Truecheck if the domain name resolves.date_time(format=None). Without argument this represents an ISO 8601 date-time. Theformatargument represents a format string forstrftime.dateandtime. These represent an ISO 8601 date and an ISO 8601 time.FormatA schema can be, in order of precedence:A class with the following properties:it has a no-argument constructor;the instances have a__validate__method with signature__validate__(self,object,name,strict)The parameters of__validate__()have the same semantics as those ofvalidate(). The return value of__validate__()should be the empty string if validation succeeds, and otherwise it should be an explanation about what went wrong.An object having a__validate__attribute with signature__validate__(object,name,strict)as above. This is for example how the wrapper schemas are implemented internally.A Python type. In that case validation is done by checking membership.A callable. Validation is done by applying the callable to the object. If applying the callable throws an exception then the corresponding message will be part of the non-validation message.Alistor atuple. Validation is done by first checking membership of the corresponding types, and then performing validation for each of the entries of the object being validated against the corresponding entries of the schema.A dictionary. Validation is done by first checking membership of thedicttype, and then performing validation for each of the items of the object being validated against the corresponding items of the schema.Aset. A set validates an object, if one of its members does.An arbitrary Python object. Validation is done by checking equality of the schema and the object, except when the schema is of typefloat, in which casemath.iscloseis used.Creating typesA cool feature ofvtjsonis that one can transform a schema into a genuine Python type viat=make_type(schema)so that validation can be done viaisinstance(object,t)The drawback, compared to usingvalidatedirectly, is that there is no feedback when validation fails. You can get it back as a console debug message via the optionaldebugargument tomake_type.
The full signature ofmake_typeismake_type(schema,name=None,strict=True,debug=False)The optionalnameargument is used to set the__name__attribute of the type. If it is not supplied thenvtjsontries to make an educated guess.Examples>>>fromvtjsonimportset_name,validate>>>schema={"fruit":{"apple","pear","strawberry"},"price":float}>>>object={"fruit":"dog","price":1.0}>>>validate(schema,object)...vtjson.ValidationError:object['fruit'](value:dog)isnotequalto'pear'andobject['fruit'](value:dog)isnotequalto'strawberry'andobject['fruit'](value:dog)isnotequalto'apple'>>>fruit=set_name({"apple","pear","strawberry"},"fruit")>>>schema={"fruit":fruit,"price":float}>>>validate(schema,object)...vtjson.ValidationError:object['fruit'](value:dog)isnotoftype'fruit'>>>object={"fruit":"apple"}>>>validate(schema,object)...vtjson.ValidationError:object['price']ismissingFor many more examples see the filetest_validate.pyin the source distribution.FAQQ: Why not just use the Python implementation ofJSON schema(seehttps://pypi.org/project/jsonschema/)?A: Various reasons.Avtjsonschema is much more concise than aJSONschema!vtjsoncan validate objects which are more general than strictlyJSON. See the introductory example above.More fundamentally, the design philosophy ofvtsjonis different. AJSONschema  is language independent and fully declarative. These are very nice properties but, this being said, declarative languages have a tendency to suffer from feature creep as they try to deal with more and more exotic use cases (e.g.css).  Avtjsonschema on the other hand leverages the versatility of the Python language. It is generally declarative, with a limited, but easily extendable set of primitives. But if more functionality is needed then it can be extended by using appropriate bits of Python code (as theordered_pairexample below illustrates). In practice this is what you will need in any case since a purely declarative language will never be able to deal with every possible validation scenario.Q: How is this different fromhttps://pypi.org/project/json-checker/?A: Good question! I discoveredjson-checkerafter I had writtenvtjson. Although the details are differentjson-checkerandvtjsonshare many of the same principles.Q: How to combine validations?A: Useintersect. For example the following schema validates positive integers but reject positive floats.schema=intersect(int,interval(0,...))More generally one may use the patternintersect(schema, more_validations)where the first argument makes sure that the object to be validated has the required layout to be an acceptable input for the later arguments. For example an ordered pair of integers can be validated using the schemadefordered_pair(o):returno[0]<=o[1]schema=intersect((int,int),ordered_pair)Or in a one linerschema=intersect((int,int),set_name(lambdao:o[0]<=o[1],"ordered_pair"))The following also works if you are content with less nice output on validation failure (try it)schema = intersect((int, int), lambda o: o[0] <= o[1])