Package: exjson
EXJSONExtended JSON Parser for PythonIntroductionEXJSON is layer over the Python Standard JSON decoder library, which implements functionality not currently supported by it while trying to keep compliant with the JSON standard as much as possible.Main FeaturesC Style Single-Line and Multi-Lines Comments.Inclusion of other JSON files from local storage or an http/https source.Included file checksum verification.Absolute and Relative Value referencing using$root,$parentand$this.Extensible Scripting.Supported Python VersionsPython 3.xInstall/Upgradepipinstallexjson--upgradeSamplesamplefile1.json{// Sample Property"name":"test file",// Sample value set with an included object"values":[/* INCLUDE "samplefile2.json" */{"value_id":"923ko30k3","value":"Another Value"}]}samplefile2.json/*INCLUDIBLE TEST FILE*/{"value_id":"93987272","value":"This Value"}Usageimportexjsonasjson# Decodesample_value_set=json.load("./samplefile1.json")# ... Do stuff with sample_value_set# Encodewithopen("./result.json")asf:f.write(json.dumps(sample_value_set))result.json{"name":"test file","values":[{"value_id":"93987272","value":"This Value"},{"value_id":"923ko30k3","value":"Another Value"}]}The#Includedirective accepts three parameters from which only the first one is required. They are pipe separated.#INCLUDE "test.json|{}|e3ae49df2030ee913f8be352999f30d7"test.json: the location of the file to include.{}: the default value to use if the file is not found and theerror_on_included_file_not_foundis set toFalse.e3ae49df2030ee913f8be352999f30d7: expected checksum. It will be used in order to verify the file validity.Additionally you are table to download the file from an http/https url and verify its checksum to prevent injection of malicious code.Include from HTTP/HTTPS{// Sample Property"name":"test file",// Sample value set with an included object"values":[/* INCLUDE "http://www.exjson.com/lab/test.json" */{"value_id":"923ko30k3","value":"Another Value"}]}Include from HTTP/HTTPS and verify checksum{// Sample Property"name":"test file",// Sample value set with an included object"values":[/* INCLUDE "http://www.exjson.com/lab/test.json|{}|e3ae49df2030ee913f8be352999f30d7" */{"value_id":"923ko30k3","value":"Another Value"}]}For more complex examples please check theunit tests.APIThe exjson API offers similar API to the one available on the Python standard JSON decoder/encoder library.load(json_file_path, encoding=None, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, object_pairs_hook=None, error_on_include_file_not_found=False, **kw)Deserializes JSON file into a dictionary.Arguments:json_file_path: main json file to be loaded.encoding: encoding codec to use when loading the file and all included files. All included files should use the same encoding.cls: if specified, it uses the provided custom JSONDecoder instance to decode the json file.See Python docs for details.object_hook: if specified, it will be called for every decoded JSON object and its value will be used instead of the defaultdict.See Python docs for details.parse_float: if specified, it will be called for everyfloatthat is decoded.See Python docs for details.parse_int: if specified, it will be called for everyintthat is decoded.See Python docs for details.parse_constant: if specified, will be called with one of the following strings: '-Infinity', 'Infinity', 'NaN'.See Python docs for details.object_pairs_hook: if specified, it will be called for every decoded JSON object with an ordered list of pairs. Its result will be used instead of the defaultdict.See Python docs for details.error_on_included_file_not_found: if set toTruean Exception is raised if an included file is not found.Supported Extended Functionality:Supports #INCLUDE directive.Supports single-line and multi-line C style commentsloads(json_string, encoding=None, cls=None, object_hook=None, parse_float=None,
parse_int=None, parse_constant=None, object_pairs_hook=None, error_on_include_file_not_found=False, includes_path=None, **kw)Deserializes JSON string into a dictionary.Arguments:json_file_path: main json file to be loaded.encoding: encoding codec to use when loading the file and all included files. All included files should use the same encoding.cls: if specified, it uses the provided custom JSONDecoder instance to decode the json file.See Python docs for details.object_hook: if specified, it will be called for every decoded JSON object and its value will be used instead of the defaultdict.See Python docs for details.parse_float: if specified, it will be called for everyfloatthat is decoded.See Python docs for details.parse_int: if specified, it will be called for everyintthat is decoded.See Python docs for details.parse_constant: if specified, will be called with one of the following strings: '-Infinity', 'Infinity', 'NaN'.See Python docs for details.object_pairs_hook: if specified, it will be called for every decoded JSON object with an ordered list of pairs. Its result will be used instead of the defaultdict.See Python docs for details.error_on_included_file_not_found: if set toTruean Exception is raised if an included file is not found.includes_path: if provided it will be used to set the root path from where the included files will be loaded. When not provided the executing python script path will be used. Please, bear in mind that#INCLUDEdirective file path is consider relative to this one.Supported Extended Functionality:Supports #INCLUDE directive.Supports single-line and multi-line C style commentsdumps(obj, skipkeys=False, ensure_ascii=True, check_circular=True,
allow_nan=True, cls=None, indent=None, separators=None,
default=None, sort_keys=False, **kw)Serializes a python object/dictionary instance into a JSON string.Arguments:obj: object instance to encode (serialize).skipkeys: If set toFalseaTypeErroris raised if the keys are not primitive types (int,str,floatorNone).See Python docs for details.ensure_ascii: If set toTrueall Incoming ASCII characters will be escaped in the output, else they will kept as-is.See Python docs for details.check_circular: If set toTruewill check all classes and dictionaries for prevent circular references in order to prevent infinite recursion.See Python docs for details.allow_nan: If set toTrue,NaN,Infinity, and-Infinitywill be encoded as such.See Python docs for details.cls:See Python docs for details.indent: If set toTruethe output json will be indented.See Python docs for details.separators: If specified, it should be a tuple listing the item and key separators to use during encoding.See Python docs for details.default:See Python docs for details.sort_keys: It set toTruethe output dictionary will be sorted by key.See Python docs for details.Supported Extended Functionality:Does not support #INCLUDE directive.Does not support comments.Features:C Style CommentsSupports C Style Comments.
Single-Line// TEST/* TEST *//*TEST*/C Style Include directiveLoads specified file from the same path where the file is being loaded.
Supports 2 syntax always enclosed in comments:/* #INCLUDE <[PropertyName:]json_file_relative_path> */PropertyName(Optional)This is the name of the JSON property that will encapsulate the included file content. This required when including a file between other properties.{"Name":"Test"/* #INCLUDES <Values:values.json> */}{"Name":"Test","Values":{"1":"Test1","2":"Test2"}}json_file_relative_pathThis is the json file name including relative path (if located in a nested folder) to the path where the main json file exists. When the main json is loaded as a string and theincludes_pathis not specified it will use the executing python script path.
If the script is not found an error will be raised.The#INCLUDEdirective arguments can be enclosed in<>or"".Access Value by Reference$root,$parentand$thisaccessor prefixes are supported. This accessors allow you to reference values from the JSON root, parent or current object even if they are being included or its value is being calculated at runtime using a function and they can be interpolated into a string without the need of enclosing characters.Sample:{"prefix":"A","first":[{"id":"A1"},{"id":"A2"},{"id":"A3"},{"id":"$root.prefix4"}],"second":"$root.prefix","third":{"test1":23,"test2":[1,2,3],"test3":{"deep1":44,"deep2":false,"deep3":"$root.secondB","deep4":"AZ-$parent.test1X"}},"fourth":{"t1":"B","t2":"$this.t1"}}Result:{"prefix":"A","first":[{"id":"A1"},{"id":"A2"},{"id":"A3"},{"id":"A4"}],"second":"A","third":{"test1":23,"test2":[1,2,3],"test3":{"deep1":44,"deep2":false,"deep3":"AB","deep4":"AZ-23X"}},"fourth":{"t1":"B","t2":"B"}}first[3]=$root.prefix4references the value ofprefixon the JSON root while interpolated in the string.second=$root.prefixreferences the value ofprefixon the JSON root.third.test3.deep3=$root.secondBreferenced the value ofsecondon the JSON root while interpolated in the string.third.test3.deep4=AZ-$parent.test1Xreferences the the value of$root.third.test1which is the parent object oftest3while interpolated in a string.fourth.t2=$this.t1references the value of$root.fourth.t1which is the parentCurrent LimitationsRelative and Absolute Value references are not supported as function parameters yet.ScriptingEXJSON supports dynamic values by using an extensible scripting engine based on python. Commonly used extension functions can be found in thescripting/extensionspackage but you can create and load your own custom extensions functions by using theregister_custom_scripting_extension.Function calls can be interpolated in strings likeAAX-$.sequence('product-sequence')ZZwhich will yieldAAX-1ZZ.Null values can be set by usingnull.Standard Functions:CryptographyMD5Generates a MD5 hash from the specified string value. When no string is provided a random long of 128 random bits is used.Signature$.md5([optional]stringvalue)Usagejson { "value": "$.md5('test message')", "value1": "$.md5()" }Resultjson { "value": "c72b9698fa1927e1dd12d3cf26ed84b2", "value1": "a1fa31838987e9a93338afb0831709d7" }SHA1Generates an SHA1 hash from the specified string value. When no string is provided a random long of 128 random bits is used.Signature$.sha1([optional]stringvalue)Usage{"value":"$.sha1('test message')","value1":"$.sha1()"}Result{"value":"35ee8386410d41d14b3f779fc95f4695f4851682","value1":"18ba91517fde987433d27dab5f49390483ac2f03"}SHA256Generates an SHA256 hash from the specified string value. When no string is provided a random long of 256 random bits is used.Signature$.sha256([optional]stringvalue)Usage{"value":"$.sha256('test message')","value1":"$.sha256()"}Result{"value":"3f0a377ba0a4a460ecb616f6507ce0d8cfa3e704025d4fda3ed0c5ca05468728","value1":"c0329b1e9a0b31d62da4855ed3730ad7104a4cd634415b38c038ccaf1db83cc3"}SHA512Generates an SHA512 hash from the specified string value. When no string is provided a random long of 512 random bits is used.Signature$.sha512([optional]stringvalue)Usage{"value":"$.sha512('test message')","value1":"$.sha512()"}Result{"value":"950b2a7effa78f51a63515ec45e03ecebe50ef2f1c41e69629b50778f11bc080002e4db8112b59d09389d10f3558f85bfdeb4f1cc55a34217af0f8547700ebf3","value1":"5ac6b81120595d01cce41f18a14440dffb435955020a682c47288eae9aacf5263a051e14bff01f688433cb575dab0ac82b6135286d0f1ced93dbc37375ffba68"}Date and TimeBy default datetime functions will return date and time inISO-8601format with explicit TZ even on UTC. ISO-8601 Zulu format can be achived by using theZformat as described in the table below:"Universal" Format CodePython Format CodeDescriptionExampledddd%AWeekday as locale’s full name.Mondayddd%aWeekday as locale’s abbreviated name.Monww%wWeekday as a zero padded number, where 0 is Sunday and 6 is Saturday.01w%-wWeekday as a decimal number, where 0 is Sunday and 6 is Saturday.1dd%dDay of the month as a zero-padded decimal number.30d%-dDay of the month as a decimal number.30MMMM%BMonth as locale’s full name.SeptemberMMM%bMonth as locale’s abbreviated name.SepMM%mMonth as a zero-padded decimal number.09M%-mMonth as a decimal number.9yyyy%YYear with century as a decimal number.2013y%yYear without century as a zero-padded decimal number.13HH%HHour (24-hour clock) as a zero-padded decimal number.19H%-HHour (24-hour clock) as a decimal number.19hh%IHour (12-hour clock) as a zero-padded decimal number.07h%-IHour (12-hour clock) as a decimal number.7tt%pLocale’s equivalent of either AM or PM.AMmm%MMinute as a zero-padded decimal number.06m%-MMinute as a decimal number.6ss%SSecond as a zero-padded decimal number.05s%-SSecond as a decimal number.5f%fMicrosecond as a decimal number, zero-padded on the left.000000zzz%ZTime zone name (empty string if the object is -00:00).-04:00z%zUTC offset in the form +HHMM or -HHMM.-04:00j%jDay of the year as a zero-padded decimal number.273jj%-jDay of the year as a decimal number.273UU%UWeek number of the year (Sunday as the first day of the week) as a zero padded decimal number. All days in a new year preceding the first Sunday are considered to be in week 0.09U%-UWeek number of the year (Sunday as the first day of the week) as a decimal number. All days in a new year preceding the first Sunday are considered to be in week 0.9WW%WWeek number of the year (Monday as the first day of the week) as a zero padded decimal number. All days in a new year preceding the first Monday are considered to be in week 0.08W%WWeek number of the year (Monday as the first day of the week) as a decimal number. All days in a new year preceding the first Monday are considered to be in week 0.8qcustomCalendar Quarter as zero padded number.04qqcustomCalendar Quarter as number.4F%cLocale’s appropriate date and time representation.Mon Sep 30 07:06:05 2013D%xLocale’s appropriate date representation.09/30/13T%XLocale’s appropriate time representation.07:06:05Z%Y-%m-%dT%H:%m:%s.%f%zZulu UTC ISO-86012018-05-25T15:05:25.120ZNo need to worry about running it on windows or linux. The universal format converter takes care of the platform specific tokens-/#.NOW()Gets the current date and time based on server timezone.Signature$.now([optional]formatstring)Usage```json
  {
    "value": "$.now('MM/dd/yyyy hh:mm tt')",
    "value1": "$.now()"
  }
  ```Result```json
  {
    "value": "05/27/2018 12:50 PM",
    "value1": "2018-05-27T12:50:05.302736-04:00"
  }
  ```NOW().ADD()Adds the specified number of time units to the current date and time based on server timezone.Unit of TimeDescriptiondayAdds or subtracts days.weekAdds or subtracts weeks.monthAdds or subtracts months.yearAdds or subtracts years.quarterAdds or subtracts quarter.Signature$.now().add([required]unitoftimecriterias...,[optional]formatstring)Usage```json
  {
    "value": "$.now().add(day=-1,'MM/dd/yyyy')",
    "value1": "$.now().add(day=-1)"
  }
  ```Result```json
  {
    "value": "05/26/2018",
    "value1": "2018-05-26T00:00:00.0000-04:00"
  }
  ```NOW().UTC()Same asnow()but returnsUTCdate and time.NOW().UTC().ADD()Same asnow().add()but returnsUTCdate and time.Sequences and IdentificationUUID4()Returns a uuid v4 hash.Signature$.uuid4()Usage```json
  {
    "value": "$.uuid4()",
  }
  ```Result```json
  {
    "value": "9f1ac603-0e7c-4d25-b73c-05ac2807d814"
  }
  ```SEQUENCE()Creates and returns one or multiple named integer sequence values. Sequences are generated per load/loads call. Sequences are destroyed once they are populated in the JSON file.Signature$.sequence([required]sequence_name,[optional]padding_pattern,[optional]step)Thepadding_patternuses thepython string formatting mini languageand thestepis defaulted to1if no value is provided.Usage```json
  {
    "value": "$.sequence('seq1',2)",
    "values": [
        "test1": "A-$.sequence('seq2', '{0:0>4}')",
        "test2": "A-$.sequence('seq2', '{0:0>4}')"
        "test3": "A-$.sequence('seq2', '{0:0>4}')"
        "test4": "A-$.sequence('seq2', '{0:0>4}')"
    ],
    "value1": "$.sequence('seq1',2)",
  }
  ```Result```json
  {
    "value": "1",
    "values": [
        "test1": "A-0001",
        "test2": "A-0002"
        "test3": "A-0003"
        "test4": "A-0004"
    ],
    "value1": "1",
  }
  ```FILE_CHECKSUM()Calculates MD5 or SHA1 checksum for the specified file. The file path should be relative to the path where the JSON file is located if loaded or included. Relative to the executing python script if using loads from a JSON string. If noalgorithmis providedmd5will be used by default.Signature$.file_checksum([required]file_path,[optional]algorithm)Supported algorithms aremd5andsha1.Usage```json
  {
    "filename": "../file.db",
    "checksum": "$.file_checksum('../file.db', 'md5')"
  }
  ```Result```json
   {
    "filename": "../file.db",
    "checksum": "ceb4c1f4b28bd59b941573d8a41f14d3"
  }
  ```How to Create a custom scripting extension function and register it at Run-Time.While standard functions can be added through issues (requests) or pull requests into the version branch you may want to use some functions that are too specific to your scenario that cannot be classified as standard. In this case you may want to register them at run-time.Standard or Custom Scripting Extension Function Example:defcustom_add(*args):result=0iflen(args)==0:result=0else:forrinargs:result=result+rreturnresultA custom scripting function should always have an*argsvariable in order to allow for an undefined number of parameters to be passed in. Parameter check can always be performed inside the function body if required. The scripting extension functio should always return a primitive type (int,str,float, etc...). Arrays and other types are not supported yet.Registration:if__name__=="__main__":exjson.register_custom_scripting_extension("add",custom_add)...exjson.load("sample.json")Usage:{"a":$.add(10,20)}Result:{"a":30}The resulting value can be accessed using the relative and absolute accessors$this,$parentand$rootfrom anywhere in the current file or an included JSON file.Life Cycle:Load JSON File ContentComments are extracted.Directives are extracted from comments.Include files (#INCLUDEdirective) content are extracted recursively (Apply steps 1 to 4).
The load process is stopped if an included file is determined to have a circular reference.Include files are merged into JSON file which included it.Script function calls are extracted and evaluated.Relative and Absolute value references are evaluated.Unit Test Requirements:EXJSON unit testing runs on thestandard Python unit test library. But I EXJSON unit test functions support automatic-generation of call diagrams from each test function execution. Because of this there is an additional dependency onPyCallGraph. Please follow the steps below in order to install this dependency on Windows, Linux or OSX.
Please bear in mind that the steps below assume you already have python 3.6+ and pip installed. Depending on how your environment is setup Python 3.xpipmay be available through an alias namedpip3.Turn on and off call graph generatorCall graph generate is controlled by an environmental variable namedGENERATE_CALL_GRAPHS. If the environmental variable does not exists the value is defaulted toFalse, no graph will be generated. If it exists and it is set toTruegraphs will be generated in thetests/callsfolder.Linux/OSXexportGENERATE_CALL_GRAPHS=TrueWindows%GENERATE_CALL_GRAPHS%=TrueUbuntuInstallGraphvizand it's development librariessudoapt-getinstallgraphvizlibgraphviz-devInstallpygraphvizpipinstallpygraphvizInstallpycallgraph.pipinstallpycallgraphAlternatively you can download thePyGraphvizwheel file from Pypi.org and installing it as described in steps 4 and 5 for Windows below.WindowsDownloadGraphviz for windowsfrom the graphiviz site.Add the Graphviz bin pathC:\Program Files (x86)\Graphviz2.38\binin your Windows path.Close and reopen your terminals so the path changes is recognized.Download thepygraphviz python 3.6 wheel.Install thepygraphvizwheel.pipinstallgraphviz-0.8.3-py2.py3-none-any.whlInstallpycallgraphpipinstallpycallgraphOSXInstallHomeBrew.Download thepygraphviz python 3.6 wheel.Install thepygraphvizwheel.pipinstallgraphviz-0.8.3-py2.py3-none-any.whlInstallpycallgraph.pipinstallpycallgraphRoad Map:Better unit testing coverage.Stabilize and resolve any bugs.Segregate and make better scripting extension documentation.Support Multi-File Serialization.For More checkIssuestagged asnext releaseordefferedfor details.