Package: jsonmerge
This Python module allows you to merge a series of JSON documents into a
single one.This problem often occurs for example when different authors fill in
different parts of a common document and you need to construct a document
that includes contributions from all the authors. It also helps when
dealing with consecutive versions of a document where different fields get
updated over time.Consider a trivial example with two documents:>>> base = {
...         "foo": 1,
...         "bar": [ "one" ],
...      }

>>> head = {
...         "bar": [ "two" ],
...         "baz": "Hello, world!"
...     }We call the document we are merging changes intobaseand the changed
documenthead. To merge these two documents usingjsonmerge:>>> from pprint import pprint

>>> from jsonmerge import merge
>>> result = merge(base, head)

>>> pprint(result, width=40)
{'bar': ['two'],
 'baz': 'Hello, world!',
 'foo': 1}As you can see, when encountering an JSON object,jsonmergeby default
returns fields that appear in eitherbaseorheaddocument. For other
JSON types, it simply replaces the older value. These principles are also
applied in case of multiple nested JSON objects.In a more realistic use case however, you might want to apply differentmerge strategiesto different parts of the document. You can telljsonmergehow to do that using a syntax based onJSON schema.If you already have schemas for your document, you can simply expand them
with some additional keywords. Apart from the custom keywords described
below,jsonmergeby default uses the schema syntax defined in theDraft
4of the JSON schema specification.You use themergeStrategyschema keyword to specify the strategy. The
default two strategies mentioned above are calledobjectMergefor objects
andoverwritefor all other types.Let’s say you want to specify that the mergedbarfield in the example
document above should contain elements from all documents, not just the
latest one. You can do this with a schema like this:>>> schema = {
...             "properties": {
...                 "bar": {
...                     "mergeStrategy": "append"
...                 }
...             }
...         }

>>> from jsonmerge import Merger
>>> merger = Merger(schema)
>>> result = merger.merge(base, head)

>>> pprint(result, width=40)
{'bar': ['one', 'two'],
 'baz': 'Hello, world!',
 'foo': 1}Another common example is when you need to keep a versioned list of values
that appeared in the series of documents:>>> schema = {
...             "properties": {
...                 "foo": {
...                     "type": "object",
...                     "mergeStrategy": "version",
...                     "mergeOptions": { "limit": 5 }
...                 }
...             },
...             "additionalProperties": False
...         }
>>> from jsonmerge import Merger
>>> merger = Merger(schema)

>>> rev1 = {
...     'foo': {
...         'greeting': 'Hello, World!'
...     }
... }

>>> rev2 = {
...     'foo': {
...         'greeting': 'Howdy, World!'
...     }
... }

>>> base = None
>>> base = merger.merge(base, rev1, merge_options={
...                     'version': {
...                         'metadata': {
...                             'revision': 1
...                         }
...                     }
...                 })
>>> base = merger.merge(base, rev2, merge_options={
...                     'version': {
...                         'metadata': {
...                             'revision': 2
...                         }
...                     }
...                 })
>>> pprint(base, width=55)
{'foo': [{'revision': 1,
          'value': {'greeting': 'Hello, World!'}},
         {'revision': 2,
          'value': {'greeting': 'Howdy, World!'}}]}Note that we use themergeOptionskeyword in the schema to supply
additional options to the merge strategy. In this case, we tell theversionstrategy to retain only 5 most recent versions of this field.We also used themerge_optionsargument to supply some options that are
specific to each call of themergemethod. Options specified this
way are applied to all invocations of a specific strategy in a schema (in
contrast tomergeOptions, which applies only to the strategy invocation
in that specific location in the schema). Options specified inmergeOptionsschema keyword override the options specified in themerge_optionsargument.Themetadataoption for theversionstrategy can contain some document
meta-data that is included for each version of the field.metadatacan
contain an arbitrary JSON object.Example above also demonstrates howjsonmergeis typically used when
merging more than two documents. Typically you start with an emptybaseand then consecutively merge differentheadsinto it.A common source of problems are documents that do not match the schema used
for merging.jsonmergeby itself does not validate input documents. It
only uses the schema to obtain necessary information to apply appropriate merge
strategies. Since the default strategies are used for parts of the
document that are not covered by the schema it’s easy to get unexpected
output without any obvious errors raised byjsonmerge.In the following example, the propertyFoo(uppercase F) does not matchfoo(lowercase f) in the schema and hence theversionstrategy is not
applied as with previous two revisions:>>> rev3 = {
...     'Foo': {
...         'greeting': 'Howdy, World!'
...     }
... }

>>> base = merger.merge(base, rev3, merge_options={
...                     'version': {
...                         'metadata': {
...                             'revision': 3
...                         }
...                     }
...                 })

>>> pprint(base, width=55)
{'Foo': {'greeting': 'Howdy, World!'},
 'foo': [{'revision': 1,
          'value': {'greeting': 'Hello, World!'}},
         {'revision': 2,
          'value': {'greeting': 'Howdy, World!'}}]}Hence it is recommended to validate the input documents against the schema
before passing them tojsonmerge. This practice is even more effective if
the schema is filled in with more information than strictly necessary forjsonmerge(e.g. adding information about types, restrict valid object
properties withadditionalProperties, etc.):>>> from jsonschema import validate
>>> validate(rev1, schema)
>>> validate(rev2, schema)
>>> validate(rev3, schema)
Traceback (most recent call last):
    ...
jsonschema.exceptions.ValidationError: Additional properties are not allowed ('Foo' was unexpected)If you care about well-formedness of your documents, you might also want to
obtain a schema for the documents that themergemethod creates.jsonmergeprovides a way to automatically generate it from a schema for
the input document:>>> result_schema = merger.get_schema()

>>> pprint(result_schema, width=80)
{'additionalProperties': False,
 'properties': {'foo': {'items': {'properties': {'value': {'type': 'object'}}},
                        'maxItems': 5,
                        'type': 'array'}}}Note that because of theversionstrategy, the type of thefoofield
changed fromobjecttoarray.Merge strategiesThese are the currently implemented merge strategies.overwriteOverwrite with the value inbasewith value inhead. Works with any
type.discardKeep the value inbase, even ifheadcontains a different value.
Works with any type.By default, ifbasedoes not contain any value (i.e. that part of the
document is undefined), the value after merge is kept undefined. This can
be changed with thekeepIfUndefoption. If this option istrue, then
the value fromheadwill be retained in this case. This is useful if
you are merging a series of documents and want to keep the value that
first appears in the series, but want to discard further modifications.appendAppend arrays. Works only with arrays.You can specify asortByRefmerge option to indicate the key that will
be used to sort the items in the array. This option can be an arbitraryJSON pointer. When resolving the pointer the root is placed at the
root of the array item. Sort order can be reversed by setting thesortReverseoption.arrayMergeByIdMerge arrays, identifying items to be merged by an ID field. Resulting
arrays have items from bothbaseandheadarrays.  Any items that
have identical an ID are merged based on the strategy specified further
down in the hierarchy.By default, array items are expected to be objects and ID of the item is
obtained from theidproperty of the object.You can specify an arbitraryJSON pointerto point to the ID of the
item using theidRefmerge option. When resolving the pointer, document
root is placed at the root of the array item (e.g. by default,idRefis
‘/id’). You can also setidRefto ‘/’ to treat an array of integers or
strings as a set of unique values.Array items inheadfor which the ID cannot be identified (e.g.idRefpointer is invalid) are ignored.You can specify an additional item ID to be ignored using theignoreIdmerge option.A compound ID can be specified by settingidRefto an array of
pointers. In that case, ifanypointer in the array is invalid for an
object inhead, the object is ignored. If using an array foridRefand ifignoreIdoption is also defined,ignoreIdmust be an array as
well.You can specify asortByRefmerge option to indicate the key that will
be used to sort the items in the array. This option can be an arbitraryJSON pointer. The pointer is resolved in the same way asidRef. Sort
order can be reversed by setting thesortReverseoption.arrayMergeByIndexMerge array items by their index in the array. Similarly toarrayMergeByIdstrategy, the resulting arrays have items from bothbaseandheadarrays. Items that occur at identical positions in both
arrays will be merged based on the strategy specified further down in the
hierarchy.objectMergeMerge objects. Resulting objects have properties from bothbaseandhead. Any properties that are present both inbaseandheadare
merged based on the strategy specified further down in the hierarchy
(e.g. inproperties,patternPropertiesoradditionalPropertiesschema keywords).TheobjClassoption allows one to request a different dictionary class
to be used to hold the JSON object. The possible values are names that
correspond to specific Python classes. Built-in names includeOrderedDict, to use the collections.OrderedDict class, ordict,
which uses the Python’s dict built-in. If not specified,dictis
used by default.Note that additional classes or a different default can be configured via
the Merger() constructor (see below).versionChanges the type of the value to an array. New values are appended to the
array in the form of an object with avalueproperty. This way all
values seen during the merge are preserved.You can add additional properties to the appended object using themetadataoption. Additionally, you can usemetadataSchemaoption to
specify the schema for the object in themetadataoption.You can limit the length of the list using thelimitoption in themergeOptionskeyword.By default, if aheaddocument contains the same value as thebase,
document, no new version will be appended. You can change this by settingignoreDupsoption tofalse.If a merge strategy is not specified in the schema,objectMergeis used
for objects andoverwritefor all other values (but see also the section
below regarding keywords that apply subschemas).You can implement your own strategies by making subclasses of
jsonmerge.strategies.Strategy and passing them to Merger() constructor
(see below).The Merger ClassThe Merger class allows you to further customize the merging of JSON
data by allowing you to:set the schema containing the merge strategy configuration,provide additional strategy implementations,set a default class to use for holding JSON object data andconfigure additional JSON object classes selectable via theobjClassmerge option.The Merger constructor takes the following arguments (all optional, except
schema):schemaThe JSON Schema that contains the merge strategy directives
provided as a JSON object.  An empty dictionary should be provided
if no strategy configuration is needed.strategiesA dictionary mapping strategy names to instances of Strategy
classes.  These will be combined with the built-in strategies
(overriding them with the instances having the same name).objclass_defThe name of a supported dictionary-like class to hold JSON data by
default in the merged result. The name must match a built-in name or one
provided in theobjclass_menuparameter.objclass_menuA dictionary providing additional classes to use as JSON object
containers.  The keys are names that can be used as values for theobjectMergestrategy’sobjClassoption or theobjclass_defargument. Each value is a function or class that produces an instance of
the JSON object container. It must support an optional dictionary-like
object as a parameter which initializes its contents.validatorclassAjsonschema.Validatorsubclass. This can be used to specify which
JSON Schema draft version will be used during merge. Some details such
as reference resolution are different between versions. By default, the
Draft 4 validator is used.Support for keywords that apply subschemasComplex merging of documents with schemas that use keywordsallOf,anyOfandoneOfcan be problematic. Such documents do not have a
well-defined type and might require merging of two values of different
types, which will fail for some strategies. In such casesget_schema()might also return schemas that never validate.Theoverwritestrategy is usually the safest choice for such schemas.If you explicitly define a merge strategy at the same level asallOf,anyOforoneOfkeyword, thenjsonmergewill use the defined strategy
and not further process any subschemas under those keywords. The
strategy however will descend as usual (e.g.objectMergewill take into
account subschemas under thepropertieskeyword at the same level asallOf).If a merge strategy is not explicitly defined and anallOforanyOfkeyword is present,jsonmergewill raise an error.If a merge strategy is not explicitly defined and anoneOfkeyword is
present,jsonmergewill continue on the branch ofoneOfthat validates
bothbaseandhead. If no branch validates, it will raise an error.You can define more complex behaviors by defining for your own strategy
that defines what to do in such cases. See docstring documentation for theStrategyclass on how to do that.Security considerationsA JSON schema document can contain$refreferences to external schemas.jsonmergeresolves URIs in these references using the mechanisms provided
by thejsonschemamodule. External references can cause HTTP or similar
network requests to be performed.Ifjsonmergeis used on untrusted input, this may lead to vulnerabilities
similar to the XML External Entity (XXE) attack.Requirementsjsonmergesupports Python 2 (2.7) and Python 3 (3.5 and newer).You needjsonschema(https://pypi.python.org/pypi/jsonschema) module
installed.InstallationTo install the latestjsonmergerelease from the Python package index:pip install jsonmergeSourceThe latest development version is available on GitHub:https://github.com/avian2/jsonmergeTo install from source, run the following from the top of the source
distribution:pip install .jsonmergeusesToxfor testing. To run the test suite run:toxTroubleshootingThe most common problem withjsonmergeis getting unexpected results from
a merge. Finding the exact reason whyjsonmergeproduced a particular
result can be complicated, especially when head and base structures are
very large. Most often the cause is a problem with either the schema or
head and base that is passed tojsonmerge, not a bug injsonmergeitself.Here are some tips for debugging issues withjsonmerge:Try to minimize the problem. Prune branches of head and base structures
that are not relevant to your issue and re-run the merge. Often just
getting a clearer view of the relevant parts exposes the problem.jsonmergeuses the standardloggingPython module to print out what
it is doing during the merge. You need to increase verbosity to DEBUG
level to see the messages.A very common mistake is misunderstanding which part of the schema
applies to which part of the head and base structures. Debug logs
mentioned in the previous point can be very helpful with that, since they
show how merge descends into hierarchies of all involved structures and
when a default strategy is used.With large head and base it’s common that parts of them are not what you
think they are. Validate your inputs against your schema using thejsonschemalibrary before passing them ontojsonmerge. Make sure your
schema is restrictive enough.Pay special attention to parts of the schema that useoneOf,anyOf,allOfkeywords. These can sometimes validate in unexpected ways.Another problem point can be$refpointers if they can cause recursion.
Using recursive schemas withjsonmergeis fine, but they can often
product unexpected results.Reporting bugs and contributing codeThank you for contributing tojsonmerge! Free software wouldn’t be
possible without contributions from users like you. However, please consider
that I maintain this project in my free time. Hence I ask you to follow
this simple etiquette to minimize the amount of effort needed to include
your contribution.Please useGitHub issuesto report bugs.Before reporting the bug, please make sure that:You’ve read this entire README file.You’ve read the Troubleshooting section of the README file.You’ve looked at existing issues if the bug has already been reported.Make sure that your report includes:Aminimal, but complete, code example that reproduces the problem,
including any JSON data required to run it. It should be something I can
copy-paste into a .py file and run.Relevant versions ofjsonmergeandjsonschema- either release number
on PyPi or the git commit hash.Copy of the traceback, in case you are reporting an unhandled exception.Example of what you think should be the correct output, in case you are
reporting wrong result of a merge or schema generation.Please useGitHub pull requeststo contribute code. Make sure that your
pull request:Passes all existing tests and includes new tests that cover added code.UpdatesREADME.rstto document added functionality.LicenseCopyright 2023, Tomaz Solc <tomaz.solc@tablix.org>The MIT License (MIT)Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the “Software”), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.