Package: ts-ids-validator
TetraScience IDS Artifact ValidatorTable of ContentsOverviewInstallationUsageValidate an IDSValidate an IDS and check for breaking changes from a previous versionValidationGenericschema.jsonelasticsearch.jsonathena.jsonexpected.jsonBreaking change validationTetra Data validationChangelogOverviewThe TetraScience IDS Artifact Validator checks that IDS artifacts follow a set of rules which
make them compatible with the Tetra Data Platform, and optionally validates that they are
compatible with additional IDS design conventions.
The validator either passes or fails with a list of the checks which led to the failure.The validator checks these files in an IDS folder:schema.jsonelasticsearch.jsonathena.jsonNote that there is a distinction between IDS Artifact validation and IDS instance validation.
This validator checks that the IDS Artifact files listed above are valid; an IDS instance validator would check that a data instance is valid against the corresponding IDS schema using a JSON Schema validator - this package does not do IDS instance validation.Versionv0.10.4InstallationInstalling this package requires access to the TetraScience JFrog Artifactory package repository.Runpip install ts-ids-validatorin the environment where you want to install the validator.
Or install using a package manager, withpoetryfor example:poetry add --dev ts-ids-validator.
Installing usingpipxmakes the CLI script available globally while still keeping dependencies in an isolated environment which may be useful when working with multiple IDSs:pipx install ts-ids-validator.This will install a script which can be run from the command line,validate-ids-artifact, which is equivalent to runningpython -m ids_validator- see below for usage.UsageRunvalidate-ids-artifact -hto see the help for this command.Validate an IDSWith the CLI interface:validate-ids-artifact--ids_dir=path/to/ids/folderThis will validate that the IDS is compatible with the Tetra Data Platform.If the schema containsproperties.@idsConventionVersionwith aconstvalue ofv1.0.0, then additional Tetra Data checks will run, validating that the IDS follows certain Tetra Data conventions such as using the standardsamplescomponent.
Note: in a future version of the validator, these@idsConventionVersionchecks will be removedValidate an IDS and check for breaking changes from a previous versionValidate with a local copy of the previous IDSvalidate-ids-artifact--ids_dir=path/to/ids/folder--previous_ids_dir=path/to/previous_ids/folder# Alternative with shorter syntaxvalidate-ids-artifact-ipath/to/ids/folder-ppath/to/previous_ids/folderAs well as running the validation of the IDS inids_dir, additional validation will happen using the previous version of the same IDS passed toprevious_ids_dir.Validate with the previous published version downloaded from the Tetra Data PlatformUse this feature to validate a local IDS against its previous version downloaded directly from the Tetra Data Platform.This will download the closest preceding version with the same namespace and slug from TDP, and use it for breaking change validation.
If no matching IDS is found, then validation will still run without breaking change validation.For example, if v1.0.0 and v2.0.0 are already published in TDP, and you are working on v2.0.1, then v2.0.0 will be used as the previous version in validation.
If you are working on v1.0.1, then v1.0.0 will be used.
If you are working on v0.1.0, then no breaking change validation will run.
As an edge case, if you are working on v2.0.0 locally, then v2.0.0 from TDP will be used as the "previous" version.To use this, the first step is to configure TDP API authentication, seehttps://developers.tetrascience.com/reference/authenticationfor instructions.There are two options for storing API configuration: environment variables, or a JSON config file.For environment variables, setTS_API_URL,TS_ORGandTS_AUTH_TOKENenvironment variables using any method, then run the validator with the--downloadflag:# Omit these environment variables if they are already set elsewhereexportTS_ORG=your-orgexportTS_API_URL=https://api.tetrascience.com/v1exportTS_AUTH_TOKEN=your-token# Command to run once environment variables are setvalidate-ids-artifact--download--ids_dirpath/to/ids/folder# Alternative with shorter syntaxvalidate-ids-artifact-d-ipath/to/ids/folderTo use a JSON config file, create a JSON file with the following structure, named for examplecfg.json(the name can be anything):{"api_url":"https://api.tetrascience.com/v1","auth_token":"your-token","org":"your-org"}Then use both the--downloadflag and the--configoption:validate-ids-artifact--download--ids_dirpath/to/ids/folder--configcfg.json# Alternative with shorter syntaxvalidate-ids-artifact-d-ipath/to/ids/folder-ccfg.jsonIgnore SSL certificate verification for TDP API usageIt is possible to ignore verifying the SSL certificate for the TDP API requests used to identify and download the previous IDS artifact.
The default functionality is to verify SSL certificates.To do this, add"ignore_ssl": trueto the JSON config file. It is false by default, which can also be set explicitly with"ignore_ssl": false.Or set the environment variableTS_IGNORE_SSLto any of the following values (case insensitive):true,True,1. It is false by default, which can also be set explicitly with one of the following values:false,False,0.When ignore SSL is True, API requests will accept any TLS certificate presented by the server, and will ignore hostname mismatches or expired certificates.
This makes requests vulnerable to man-in-the-middle (MitM) attacks.
Setting it to True may be useful during local development or testing.
This is handled by therequestspackage, withignore_ssl=Truecorresponding toverify=False, documentedhere.ValidationThis is an overview of the validation which is run by this IDS Artifact validator.Genericschema.json,expected.json,elasticsearch.jsonandathena.jsonmust be present in the IDS artifact.
The validation of each of them is described below.schema.jsonschema.jsonis validated against the JSON Schema draft 7 specification usingjsonschema'sValidator.check_schemamethod.
This ensures that the JSON Schema vocabulary is being used correctly, including some format validation like the root$idand$schemabeing valid URIs.Additional validation makes sure the schema meets other TDP requirements:The top-level IDS object must contain properties"@idsType","@idsVersion"and"@idsNamespace"."@idsType"must be a constant with a value consisting of a "v" followed by a validsemantic version, such as"v1.0.0"."$id"at the root of the schema must follow the formathttps://ids.tetrascience.com/<namespace>/<type>/<version>/schema.jsonwhere namespace, type and version are the constant values of the@idsNamespace,@idsTypeand@idsVersionproperties."$schema"at the root of the schema must be the URI"http://json-schema.org/draft-07/schema#": draft 7 is the version of JSON Schema supported by TDP.All objects must haveadditional_propertiesset tofalseAll properties must have a valid JSON Schema typeAn object'srequiredproperties must be defined in the object'spropertiesdefinition.datacubesmust have a schema which is compatible with the platform's requirements:datacubestype must be an array of objects.The propertiesname,dimensionsandmeasuresare present andrequired.minItems == maxItemsfordimensionsandmeasures.measures.value:Contains nested arrays so that it is anN-dimensional array, withNbeing the number ofdimensions.The innermost type is either"number",["number", "null"],"string", or["string", "null"](or equivalent).dimensions.scalemust be an array ofnumbers.All properties in the schema must have valid names for mapping to Athena:No leading underscoresNo more than 1 consecutive underscore anywhere in the propertyNo special characters (allowed characters follow the regular expression[a-zA-Z0-9_]).Exceptions to this rule are@idsNamespace,@idsType,@idsVersion, and@idsConventionVersionat the root level of the schema, and@linkanywhere in the schema.No two properties inschema.jsonmay normalize to the same Athena column name.For example, a property callednameinside an objectpersonwill correspond to an Athena column ofperson_name.
This means there cannot be another property calledperson_namedefined at the same level asperson, becauseperson_namewould clash withperson.namewhen mapping the data to Athena.No property may have the nameuuidorparent_uuidbecause these are reserved for use as Athena column names in TDP.elasticsearch.jsonelasticsearch.json's"mapping"object must match a default ElasticSearch mapping generated fromschema.json. The diff between the expected and actualelasticsearch.jsonis shown in the validator output, which can be used to updateelasticsearch.json.athena.jsonPartition paths must correspond to valid properties inschema.jsonPartition paths cannot point to properties anywhere inside an arraypartition.namecannot clash with any normalized property name fromschema.json. For example, whenschema.jsonproperties are mapped to Athena, a propertynameinside an objectpersongets a normalized Athena column name ofperson_name, meaningpartition.namecannot beperson_namebecause it would clash with theperson_nameAthena column.expected.jsonexpected.jsonmust be a valid instance ofschema.jsonusing a JSON Schema draft 7 validator.Breaking change validationBreaking change validation runs if a previous version of the IDS is passed using--previous_ids_diror--download, seevalidating breaking changes.A change to an IDS artifact is a breaking change if:Athena tables would be changed.IDS instances which were valid against the previous IDS version are not valid against the new version (excluding the top-level@idsVersionproperty, whoseconstvalue changes between every IDS version).The breaking change checks run by the validator are:Check that the two IDS artifacts have the same namespace and type, fail if they don't.The versions must either be equal (for a documentation-only change), or the new version must be a major/minor/patch bump of the previous version.Determine whether the two versions may have breaking changes, according to Semantic Versioning. For example, if the previous IDS wasv1.0.0and the current IDS isv1.1.0, then there should be no breaking changes. If the current IDS werev2.0.0instead, then there may be breaking changes.If breaking changes are not allowed according to the version change, validate that no breaking changes are included in the current version of the IDS:schema.json:All property names and paths must be the same: no properties added, removed or renamed.The type of each property must be the same, or have"null"added if it is a primitive type (any type other than"array"or"object"). For example, a change from"type": "string"to"type": ["string", "null"]is not considered a breaking change. Note that the opposite, removing"null"from the type, is a breaking change.The list of required fields for each object must either stay the same or have items removed. For example, a change from"required": ["name", "kind"]to"required": ["name"]is not a breaking change. Adding properties torequiredis a breaking change.athena.json:Any change to athena.json is a breaking change (not including file formatting changes such as changing whitespace).Note: When updating an invalid IDS, such as one which is missing a required artifact file, this breaking change validation may lead to an error because it expects the previous IDS to be valid.
In this case, do not use the--previous_ids_diror--downloadoptions which both enable breaking change validation: just validate the newly updated version of the IDS and consider using a major version bump so that any problems caused by the previous IDS being invalid will not affect the updated IDS.For example, if the previous version of the IDS is missingelasticsearch.json, an exception will be raised during validation.Tetra Data validationThese checks validate that Tetra Data conventions are being followed in this IDS'sschema.json.
These checks are enabled by including a property@idsConventionVersionwith aconstvalue ofv1.0.0inschema.json.Note that this Tetra Data specific validation will be removed from this package in a future version, so that it will only validate Tetra Data Platform requirements for IDS Artifacts.Properties of objects shouldn't begin with the same string as the object's name, for example an object calledmethodshouldn't contain a property calledmethod_name.Property names should use snake case: entirely lower-case or numeric characters separated by single consecutive underscores.related_files.pointer'sfileIdandfileKeyproperties are excluded from this check. So are any properties whose name starts with@.For standard Tetra Data components, there is validation that the schema matches the expected component structure, including property names, types andrequiredproperties. This applies tosamples,users,systemsandrelated_files.
The validator output will explain any differences from the expected component structure if this check fails.Changelogv0.10.4Update the--downloadoption to accept anignore_sslconfiguration (either an"ignore_ssl"key in the JSON config, or theTS_IGNORE_SSLenvironment variable).
This makes the API configuration the same as whatts-sdkaccepts.
See "Ignore SSL certificate verification" in the Readme for more information.v0.10.3Update validation so it is mandatory to includesamplesandusersfields along with specific definitions of each at the top level of Tetra Data IDSs to match documented conventions. This enables downstream use cases which depend on these fields always being present, even if they are not populated by any Tetra Data protocol.Add ability to download previous IDS artifacts from the Tetra Data Platform for versioning validation, as an alternative to the--previous_ids_dirCLI option.
This adds a--download/-dflag to the CLI, to specify that the previous IDS artifact should be downloaded from TDP.
API configuration can be set using environment variables, or the--config/-cCLI option which takes a path to an API configuration JSON file.
See the README for details.v0.10.2Improve readability of versioning requirements and breaking change validation in validation output.v0.10.1Remove unused dependencies which caused installation to fail in some Python versions.v0.10.0Update how to use this package: add a CLI scriptvalidate-ids-artifactwhich is equivalent to the previous approach of runningpython -m ids_validator. This previous approach still works as before but may be deprecated in a future version, so switching to the CLI script is recommended.Add validation that property names do not contain special characters.Add validation for breaking changes between IDS versions which would be incompatible with the Tetra Data PlatformAdd the new CLI argument--previous_ids_dir(which may be omitted) which is the folder containing the previous version of the same IDS namespace and type. Without this CLI argument, breaking change validation does not run.Add validation thatexpected.jsonis a valid instance ofschema.jsonusing a JSON Schema validator.Remove--versionCLI argument because this information can be retrieved from the IDS artifact being validated.v0.9.16Remove the upper bound of what propertiessamplesmay contain for Tetra Data validation. This means thesamplesschema can now include properties other than the ones in thesamplesTetra Data component, such as primary and foreign key fields.v0.9.15Limit version oftyping-extensionsin dependencies to avoid a bug which causes the validator to always fail in Python 3.10 or later.v0.9.14Updatesamples[*]check to optionally allow for it to contain a propertypk_samplesof type"string".v0.9.13related_filesis no longer checked against annotation fields like "description".v0.9.12Update check forsamples[*].labels[*].source.nametype: previously the type was
required to be"string", now it is required to be either["string", "null"]or"string", with"string"leading to a deprecation warning. This change makes thissourcedefinition the same assamples[*].properties[*].sourcein a
backward-compatible way.v0.9.11Fix bug inAthenaCheckerto allow root level IDS properties as partition paths.UpdateTypeCheckerto catch errors related to undefined/misspelledtypekey.Updatejsonschemaversion to fix package installation errorv0.9.10ModifyV1SnakeCaseCheckerto ignore checks for keys present indefinitionsobject.Add temporary allowance for@linkin*.propertiesv0.9.9Lockjsonschemaversion in requirements.txtv0.9.8ModifyRulesCheckerto log missing and extra propertiesv0.9.7Allow properties withconstvalues to have non-nullabletypev0.9.6Add checker classes for generic validationAdd checker classes for v1.0.0 convention validation