Package: genson
GenSONGenSONis a powerful, user-friendlyJSON Schemagenerator built in Python.NoteThis isnotthe Python equivalent of theJava Genson library. If you are coming from Java and need to create JSON objects in Python, you wantPython’s builtin json library.)GenSON’s core function is to take JSON objects and generate schemas that describe them, but it is unique in its ability tomergeschemas. It was originally built to describe the common structure of a large number of JSON objects, and it uses its merging ability to generate a single schema from any number of JSON objects and/or schemas.GenSON’s schema builder follows these three rules:Everyobject it is given must validate under the generated schema.Anyobject that is valid underanyschema it is given must also validate under the generated schema. (there is one glaring exception to this, detailedbelow)The generated schema should be as strict as possible given the first 2 rules.JSON Schema ImplementationGenSONis compatible with JSON Schema Draft 6 and above.It is important to note that GenSON uses only a subset of JSON Schema’s capabilities. This is mainly because it doesn’t know the specifics of your data model, and it tries to avoid guessing them. Its purpose is to generate the basic structure so that you can skip the boilerplate and focus on the details of the schema.Currently, GenSON only deals with these keywords:"$schema""type""items""properties""patternProperties""required""anyOf"You should be aware that this limited vocabulary could cause GenSON to violate rules 1 and 2. If you feed it schemas with advanced keywords, it will just blindly pass them on to the final schema. Note that"$ref"andidare also not supported, so GenSON will not dereference linked nodes when building a schema.Installation$pipinstallgensonCLI ToolThe package includes agensonexecutable that allows you to access this functionality from the command line. For usage info, run with--help:$genson--helpusage: genson [-h] [--version] [-d DELIM] [-e ENCODING] [-i SPACES]
              [-s SCHEMA] [-$ SCHEMA_URI]
              ...

Generate one, unified JSON Schema from one or more JSON objects and/or JSON
Schemas. Compatible with JSON-Schema Draft 4 and above.

positional arguments:
  object                Files containing JSON objects (defaults to stdin if no
                        arguments are passed).

optional arguments:
  -h, --help            Show this help message and exit.
  --version             Show version number and exit.
  -d DELIM, --delimiter DELIM
                        Set a delimiter. Use this option if the input files
                        contain multiple JSON objects/schemas. You can pass
                        any string. A few cases ('newline', 'tab', 'space')
                        will get converted to a whitespace character. If this
                        option is omitted, the parser will try to auto-detect
                        boundaries.
  -e ENCODING, --encoding ENCODING
                        Use ENCODING instead of the default system encoding
                        when reading files. ENCODING must be a valid codec
                        name or alias.
  -i SPACES, --indent SPACES
                        Pretty-print the output, indenting SPACES spaces.
  -s SCHEMA, --schema SCHEMA
                        File containing a JSON Schema (can be specified
                        multiple times to merge schemas).
  -$ SCHEMA_URI, --schema-uri SCHEMA_URI
                        The value of the '$schema' keyword (defaults to
                        'http://json-schema.org/schema#' or can be specified
                        in a schema with the -s option). If 'NULL' is passed,
                        the "$schema" keyword will not be included in the
                        result.NoteThe--encodingoption is only available in Python 3.GenSON Python APISchemaBuilderis the basic schema generator class.SchemaBuilderinstances can be loaded up with existing schemas and objects before being serialized.>>>fromgensonimportSchemaBuilder>>>builder=SchemaBuilder()>>>builder.add_schema({"type":"object","properties":{}})>>>builder.add_object({"hi":"there"})>>>builder.add_object({"hi":5})>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'object','properties':{'hi':{'type':['integer','string']}},'required':['hi']}>>>print(builder.to_json(indent=2)){"$schema":"http://json-schema.org/schema#","type":"object","properties":{"hi":{"type":["integer","string"]}},"required":["hi"]}SchemaBuilderAPI__init__(schema_uri=None)param schema_uri:value of the$schemakeyword. If not given, it will use the value of the first available$schemakeyword on an added schema or else the default:'http://json-schema.org/schema#'. A value ofFalseorNonewill direct GenSON to leave out the"$schema"keyword.add_schema(schema)Merge in a JSON schema. This can be adictor anotherSchemaBuilderobject.param schema:a JSON SchemaNoteThere is no schema validation. If you pass in a bad schema,
you might get back a bad schema.add_object(obj)Modify the schema to accommodate an object.param obj:any object or scalar that can be serialized in JSONto_schema()Generate a schema based on previous inputs.rtype:dictto_json()Generate a schema and convert it directly to serialized JSON.rtype:str__eq__(other)Check for equality with anotherSchemaBuilderobject.param other:anotherSchemaBuilderobject. Other types are accepted, but will always returnFalseSchemaBuilder object interactionSchemaBuilderobjects can also interact with each other:You can pass one schema directly to another to merge them.You can compare schema equality directly.>>>fromgensonimportSchemaBuilder>>>b1=SchemaBuilder()>>>b1.add_schema({"type":"object","properties":{..."hi":{"type":"string"}}})>>>b2=SchemaBuilder()>>>b2.add_schema({"type":"object","properties":{..."hi":{"type":"integer"}}})>>>b1==b2False>>>b1.add_schema(b2)>>>b2.add_schema(b1)>>>b1==b2True>>>b1.to_schema(){'$schema':'http://json-schema.org/schema#','type':'object','properties':{'hi':{'type':['integer','string']}}}Seed SchemasThere are several cases where multiple valid schemas could be generated from the same object. GenSON makes a default choice in all these ambiguous cases, but if you want it to choose differently, you can tell it what to do using aseed schema.Seeding ArraysFor example, suppose you have a simple array with two items:['one',1]There are always two ways for GenSON to interpret any array: List and Tuple. Lists have one schema for every item, whereas Tuples have a different schema for every array position. This is analogous to the (now deprecated)merge_arraysoption from version 0. You can read more about JSON Schemaarray validation here.List Validation{"type":"array","items":{"type":["integer","string"]}}Tuple Validation{"type":"array","items":[{"type":"integer"},{"type":"string"}]}By default, GenSON always interprets arrays using list validation, but you can tell it to use tuple validation by seeding it with a schema.>>>fromgensonimportSchemaBuilder>>>builder=SchemaBuilder()>>>builder.add_object(['one',1])>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'array','items':{'type':['integer','string']}}>>>builder=SchemaBuilder()>>>seed_schema={'type':'array','items':[]}>>>builder.add_schema(seed_schema)>>>builder.add_object(['one',1])>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'array','items':[{'type':'string'},{'type':'integer'}]}Note that in this case, the seed schema is actually invalid. You can’t have an empty array as the value for anitemskeyword. But GenSON is a generator, not a validator, so you can fudge a little. GenSON will modify the generated schema so that it is valid, provided that there aren’t invalid keywords beyond the ones it knows about.Seeding patternPropertiesSupport forpatternPropertiesis new in version 1; however, since GenSON’s default behavior is to only useproperties, this powerful keyword can only be utilized with seed schemas. You will need to supply anobjectschema with apatternPropertiesobject whose keys are RegEx strings. Again, you can fudge here and set the values to null instead of creating valid subschemas.>>>fromgensonimportSchemaBuilder>>>builder=SchemaBuilder()>>>builder.add_schema({'type':'object','patternProperties':{r'^\d+$':None}})>>>builder.add_object({'1':1,'2':2,'3':3})>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'object','patternProperties':{'^\\d+$':{'type':'integer'}}}There are a few gotchas you should be aware of here:GenSON is written in Python, so it uses thePython flavor of RegEx.GenSON still preferspropertiestopatternPropertiesif a property already exists that matches one of your patterns, the normal property will be updated,notthe pattern property.If a key matches multiple patterns, there isno guaranteeof which one will be updated.ThepatternPropertiesdocs themselves have some more useful pointers that can save you time.Typeless SchemasIn version 0, GenSON did not accept a schema without a type, but in order to be flexible in the support of seed schemas, support was added for version 1. However, GenSON violates rule #2 in its handling of typeless schemas. Any object will validate under an empty schema, but GenSON incorporates typeless schemas into the first-available typed schema, and since typed schemas are stricter than typless ones, objects that would validate under an added schema will not validate under the result.CustomizingSchemaBuilderYou can extend theSchemaBuilderclass to add in your own logic (e.g. recordingminimumandmaximumfor a number). In order to do this, you need to:Create a customSchemaStrategyclass.Create aSchemaBuildersubclass that includes your customSchemaStrategyclass(es).Use your customSchemaBuilderjust like you would the stockSchemaBuilder.SchemaStrategyClassesGenSON uses the Strategy Pattern to parse, update, and serialize different kinds of schemas that behave in different ways. There are severalSchemaStrategyclasses that roughly correspond to different schema types. GenSON maps each node in an object or schema to an instance of one of these classes. Each instance stores the current schema state and updates or returns it when required.You can modify the specific ways these classes work by extending them. You can inherit from any existingSchemaStrategyclass, thoughSchemaStrategyandTypedSchemaStrategyare the most useful base classes. You should callsuperand pass along all arguments when overriding any instance methods.The documentation below explains the public API and what you need to extend and override at a high level. Feel free to explorethe codeto see more, but know that the public API is documented here, and anything else you depend on could be subject to change. AllSchemaStrategysubclasses maintain the public API though, so you can extend any of them in this way.SchemaStrategyAPI[class constant]KEYWORDSThis should be a tuple listing all of the JSON-schema keywords that this strategy knows how to handle. Any keywords encountered in added schemas will be be naively passed on to the generated schema unless they are in this list (or you override that behavior into_schema).When adding keywords to a newSchemaStrategy, it’s best to splat the parent class’sKEYWORDSinto the new tuple.[class method]match_schema(cls, schema)Returntrueif this strategy should be used to handle the passed-in schema.param schema:a JSON Schema indictformrtype:bool[class method]match_object(cls, obj)Returntrueif this strategy should be used to handle the passed-in object.param obj:any object or scalar that can be serialized in JSONrtype:bool__init__(self, node_class)Override this method if you need to initialize an instance variable.param node_class:This param is not part of the public API. Pass it along tosuper.add_schema(self, schema)Override this to modify how a schema is parsed and stored.param schema:a JSON Schema indictformadd_object(self, obj)Override this to change the way a schemas are inferred from objects.param obj:any object or scalar that can be serialized in JSONto_schema(self)Override this method to customize how a schema object is constructed from the inputs. It is suggested that you invokesuperas the basis for the return value, but it is not required.rtype:dictNoteThere is no schema validation. If you return a bad schema from this method,SchemaBuilderwill output a bad schema.__eq__(self, other)When checking forSchemaBuilderequality, strategies are matched using__eq__. The default implementation uses a simple__dict__equality check.Override this method if you need to override that behavior. This may be useful if you add instance variables that aren’t relevant to whether two SchemaStrategies are considered equal.rtype:boolTypedSchemaStrategyAPIThis is an abstract schema strategy for making simple schemas that only deal with thetypekeyword, but you can extend it to add more functionality. Subclasses must define the following two class constants, but you get the entireSchemaStrategyinterface for free.[class constant]JS_TYPEThis will be the value of thetypekeyword in the generated schema. It is also used to match any added schemas.[class constant]PYTHON_TYPEThis is a Python type or tuple of types that will be matched against an added object usingisinstance.ExtendingSchemaBuilderOnce you have extendedSchemaStrategytypes, you’ll need to create aSchemaBuilderclass that uses them, since the defaultSchemaBuilderonly incorporates the default strategies. To do this, extend theSchemaBuilderclass and define one of these two constants inside it:[class constant]EXTRA_STRATEGIESThis is the standard (and suggested) way to add strategies. Set it to a tuple of all your new strategies, and they will be added to the existing list of strategies to check. This preserves all the existing functionality.Note that order matters. GenSON checks the list in order, so the first strategy has priority over the second and so on. AllEXTRA_STRATEGIEShave priority over the default strategies.[class constant]STRATEGIESThis clobbers the existing list of strategies and completely replaces it. Set it to a tuple just like forEXTRA_STRATEGIES, but note that if any object or schema gets added that your exhaustive list of strategies doesn’t know how to handle, you’ll get an error. You should avoid doing this unless you’re extending most or all existing strategies in some way.Example:MinNumberHere’s some example code creating a number strategy that tracks theminimum numberseen and includes it in the output schema.NoteThis example is written in Python 3.3+. Custom strategies also work in Python 2.7, but you need different syntax (superarguments & no splattingKEYWORDS).fromgensonimportSchemaBuilderfromgenson.schema.strategiesimportNumberclassMinNumber(Number):# add 'minimum' to list of keywordsKEYWORDS=(*Number.KEYWORDS,'minimum')# create a new instance variabledef__init__(self,node_class):super().__init__(node_class)self.min=None# capture 'minimum's from schemasdefadd_schema(self,schema):super().add_schema(schema)ifself.minisNone:self.min=schema.get('minimum')elif'minimum'inschema:self.min=min(self.min,schema['minimum'])# adjust minimum based on the datadefadd_object(self,obj):super().add_object(obj)self.min=objifself.minisNoneelsemin(self.min,obj)# include 'minimum' in the outputdefto_schema(self):schema=super().to_schema()schema['minimum']=self.minreturnschema# new SchemaBuilder class that uses the MinNumber strategy in addition# to the existing strategies. Both MinNumber and Number are active, but# MinNumber has priority, so it effectively replaces Number.classMinNumberSchemaBuilder(SchemaBuilder):""" all number nodes include minimum """EXTRA_STRATEGIES=(MinNumber,)# this class *ONLY* has the MinNumber strategy. Any object that is not# a number will cause an error.classExclusiveMinNumberSchemaBuilder(SchemaBuilder):""" all number nodes include minimum, and only handles number """STRATEGIES=(MinNumber,)Now that we have the MinNumberSchemaBuilder class, let’s see how it works.>>>builder=MinNumberSchemaBuilder()>>>builder.add_object(5)>>>builder.add_object(7)>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'integer','minimum':5}>>>builder.add_object(-2)>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'integer','minimum':-2}>>>builder.add_schema({'$schema':'http://json-schema.org/schema#','type':'integer','minimum':-7})>>>builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'integer','minimum':-7}Note that the exclusive builder is much more particular.>>>builder=MinNumberSchemaBuilder()>>>picky_builder=ExclusiveMinNumberSchemaBuilder()>>>picky_builder.add_object(5)>>>picky_builder.to_schema(){'$schema':'http://json-schema.org/schema#','type':'integer','minimum':5}>>>builder.add_object(None)# this is fine>>>picky_builder.add_object(None)# this failsgenson.schema.node.SchemaGenerationError:Couldnotfindmatchingschematypeforobject:NoneCompatibilityGenSON has been tested and verified under the following Python versions:Python 2.7.13Python 3.3.5Python 3.4.5Python 3.5.4Python 3.6.2Python 3.7.3Python 3.8.2ContributingWhen contributing, please follow these steps:Clone the repo and make your changes.Make sure your code has test cases written against it.Make sure all the tests pass.Lint your code withFlake8.Ensure the docs are accurate.Add your name to the list of contributers.Submit a Pull Request.TestsTests are written inunittest. You can run them all easily with the included executablebin/test.py.$bin/test.pyYou can also invoke individual test suites:$bin/test.py--test-suitetest.test_gen_singlePotential Future FeaturesThe following are extra features under consideration.recognize every validation keyword and ignore any that don’t applyoption to set error levelcustom serializer pluginslogical support for more keywords:enumminimum/maximumminLength/maxLengthminItems/maxItemsminProperties/maxPropertiesadditionalItemsadditionalPropertiesformat&pattern$ref&idHistory1.2.2add__version__attr to module &--versionoption to CLI tooladd--encodingoption to CLI tool that overrides default file encoding (fixes #47)1.2.1exposeSchemaStrategy.__eq__()for extensionadd support for Python 3.8update Trove classifiersBugfix:SchemaBuilder.__eq__()wasn’t matching the$schemakeyword correctlyBugfix: only activate emptyrequiredoption whenrequiredis actualy empty1.2.0SchemaStrategiesare now extendable, enabling customSchemaBuilderclasses.optimize__eq__logic1.1.0add support for Python 3.7drop support for Python 3.3drop support for JSON-Schema Draft 4 (because it doesn’t allow emptyrequiredarrays)Bugfix: preserve emptyrequiredarrays (fixes #25)Bugfix: handle nestedanyOfkeywords (fixes #35)1.0.2add support forlongintegers in Python 2.7update test-skipping decorator to use standard version requirement strings1.0.1Bugfix: seeding an object schema with a"required"keyword caused an errorDocs: fix mislabeled method1.0.0This version was a total overhaul. The main change was to split Schema into three separate classes, making it simpler to add more complicated functionality by having different generator types to handle the different schema types.SchemaNodeto manage the tree structureSchemaGeneratorfor the schema generation logicSchemaBuilderto manage the public APIInterface ChangesSchemaBuilderis the newSchemato_dict()is now calledto_schema()To make the transition easier, there is still aSchemaclass that wrapsSchemaBuilderwith a backwards-compatibility layer, but you will trigger aPendingDeprecationWarningif you use it.Seed SchemasThemerge_arraysoption has been removed in favor of seed schemas. You can now seed specific nodes as list or tuple instead of setting a global option for every node in the schema tree.You can also now seed object nodes withpatternProperties, which was a highly requested feature.Other Changesinclude"$schema"keywordaccept schemas without"type"keyworduse"anyOf"keyword to help combine schemasaddSchemaGenerationErrorfor better error handlingempty"properties"and"items"are not included in generated schemasgensonexecutablenew--schema-urioptionauto-detect object boundaries by default0.2.3Docs: add installation instructions0.2.2Docs: Python 3.6 is now explicitly tested and listed as compatible.0.2.1Bugfix:add_schemafailed when adding list-style array schemasBugfix: typo in readme0.2.0Bugfix: Options were not propagated down to subschemas.Bugfix: Empty arrays resulted in invalid schemas because it still included anitemsproperty.Bugfix:itemswas being set to a list even whenmerge_arrayswas set toTrue. This resulted in overly permissive schemas becauseitemsare matched optionally by default.Improvement: Positional Array Matching - In order to be more consistent with the way JSON Schema works, the alternate tomerge_arraysis no longer never to merge list items, but instead to merge them based on their position in the list.Improvement: Schema Incompatibility Warning - A schema incompatibility used to cause a fatal error with a nondescript warning. The message has been improved and it has been reduced to a warning.0.1.0 (2014-11-29)Initial releaseCreditsGenSONis written and maintained byJon Wolverton.ContributorsBrad SokolDavid KayHeiho1YehudaCorsia