Package: reasoner-validator
Reasoner ValidatorThis package provides software methods to Translator components (e.g. Knowledge Providers and Autonomous Relay Agents) usingany versionof theTranslator Reasoner API (TRAPI)and theBiolink Model.Seethe full documentationandthe contributor guidelines.Using the PackagePython DependencyThe Reasoner Validator now requires Python 3.9 or later (some library dependencies now force this).Installing the ModuleThe module may be installed directly from pypi.org using (Python 3)piporpip3, namely:pipinstallreasoner-validatorIf you want to install it with the extra dependencies for using it as a web service, you can use:pipinstall"reasoner-validator[web]"Installing and working with the module locally from sourceAs of release 3.1.6, this project uses thepoetry dependency managementtool to orchestrate its installation and dependencies.Afterinstalling poetryand cloning the project, the poetry installation may be run:gitclonehttps://github.com/NCATSTranslator/reasoner-validator.gitcdreasoner-validator
poetryinstallTo develop this package, install with all extras dependencies using:poetryinstall--all-extrasRunning Validation against an ARS UUID Result(*) or using a Local TRAPI Request QueryA local scripttrapi_validator.pyis available to run TRAPI Response validation against either a PK (UUID)
indexed query result of the Biomedical Knowledge Translator "Autonomous Relay System" (ARS), a local JSON Response
text file or a locally triggeredad hocquery Request against an directly specified TRAPI endpoint.Note that it is best run within apoetry shellcreated bypoetry install.For script usage, type:./trapi_validator.py--help(*) Thank you Eric Deutsch for the prototype code for this scriptRunning testsTo run the test locally install with thedevdependencies group if not already done:poetryinstall--extrasdevRun the tests with coverage report:poetryrunpytest--covNote thatpoetry automatically uses any existing virtual environment, but you can otherwise also enter the one that is created by poetry by default:poetryshell# run your commands, e.g. the web service moduleexit# exit the poetry shellThe use of the Poetry shell command allows for running of the tests without thepoetry runprefix. We will continue in this manner.%poetryshell(reasoner-validator-py3.9)%pytest--covRun the tests with detailed coverage report in a HTML page:pytest--cov--cov-reporthtmlServe the report onhttp://localhost:3000:python-mhttp.server3000--directory./htmlcovBuilding the Documentation LocallyAll paths here are relative to the root project directory.First install the documentation-specific dependencies.poetryinstall--extrasdocs# or poetry install --all-extrasThe validation codes MarkDown file should first be regenerated if needed (i.e. if it was revised):cdreasoner_validator
python./validation_codes.pyThen build the documentation locally:cd../docs
makehtmlThe resultingindex.htmland related pages describing the programmatic API are now available for viewing within the docs subfolder _build/html.Validation Run as a Web ServiceThe Reasoner Validator is available wrapped as a simple web service.  The service may be run directly or as a Docker container.APIThe web service has a single POST endpoint/validatetaking a simple JSON request body, as follows:{"trapi_version":"1.4.1","biolink_version":"3.5.0","target_provenance":{"ara_source":"infores:aragorn","kp_source":"infores:panther","kp_source_type":"primary"},"strict_validation":true,"response":"{<some full JSON object of a TRAPI query Response...>}"}The request body consists of JSON data structure with two top level tag:Anoptionaltrapi_versiontag can be given a value of the TRAPI version against which the message will be validated, expressed as a SemVer string (defaults to 'latest' if omitted; partial SemVer strings are resolved to their 'latest' minor and patch releases). This value may also be a GitHub branch name (e.g. 'master').Anoptionalbiolink_versiontag can be given a value of the Biolink Model version against which the message knowledge graph semantic contents will be validated, expressed as a SemVer string (defaults to 'latest' Biolink Model Toolkit supported version, if omitted).Anoptionaltarget_provenancewith an object dictionary (example shown) specifying the ARA and KP infores-specified knowledge sources expected to be recovered in the TRAPI query results (specified by infores CURIE) and the expected KP provenance source type, i.e. 'primary' implies that the KP is tagged as a 'biolink:primary_knowledge_source'. Optional in that the root "target_provenance" or any of the subsidiary tags may be omitted (default to None)Anoptionalstrict_validationflag (default: None or 'false'). If 'true' then follow strict validation rules, such as treating as 'error' states the use ofcategory,predicateandattribute_type_idthat are of typeabstractormixinas errors.Amandatorymessagetag should have as its value the complete JSON TRAPIResponseto be validated (See the example below)Running the Web Service DirectlyFirst install the web-specific dependencies.poetryinstall--extrasweb# or poetry install --all-extrasThe service may be run directly as a Python module. The web services module may be directly run, as follows.python-mapi.mainGo tohttp://localhost/docsto see the service documentation and to use the simple UI to input TRAPI messages for validation.Typical OutputAs an example of the kind of output to expect, if one posts the following TRAPI Response JSON data structure to the/validateendpoint:{"trapi_version":"1.4.1","biolink_version":"3.5.0","response":{"message":{"query_graph":{"nodes":{"type-2 diabetes":{"ids":["MONDO:0005148"]},"drug":{"categories":["biolink:Drug"]}},"edges":{"treats":{"subject":"drug","predicates":["biolink:treats"],"object":"type-2 diabetes"}}},"knowledge_graph":{"nodes":{"MONDO:0005148":{"name":"type-2 diabetes"},"CHEBI:6801":{"name":"metformin","categories":["biolink:Drug"]}},"edges":{"df87ff82":{"subject":"CHEBI:6801","predicate":"biolink:treats","object":"MONDO:0005148"}}},"results":[{"node_bindings":{"type-2 diabetes":[{"id":"MONDO:0005148"}],"drug":[{"id":"CHEBI:6801"}]},"edge_bindings":{"treats":[{"id":"df87ff82"}]}}]},"workflow":[{"id":"annotate"}]}}one should typically get a response body something like the following JSON validation result back:{"trapi_version":"1.4.1","biolink_version":"3.2.1","messages":{"_comment":"some categories of messages may be absent, hence, empty dictionaries","critical":{},"errors":{"error.knowledge_graph.node.category.missing":{"_comment":"source scope of the validation error ('global' or some knowledge source path string)","global":{"_comment":"this message template does not have any additional parameters other than identifier hence it just has the unique identifier  value as a dictionary key, with associated value None","MONDO:0005148":null}}},"warnings":{"_comment":"validation code","warning.knowledge_graph.node.unmapped_prefix":{"global":{"_comment":"template identifier field value","CHEBI:6801":[{"_comment":"additional message template field values, if applicable","categories":"['biolink:Drug']"}]}}},"information":{},}}To minimize redundancy in validation messages, messages are uniquely indexed in dictionaries at two levels:the (codes.yaml recorded) dot-delimited validation code path stringfor messages with templated parameters, by a mandatory 'identifier' field (which is expected to exist as a field in a template if such template has one or more parameterized fields)OpenTelemetry and JaegerThe web service may be monitored for OpenTelemetry by setting an environment variableTELEMETRY_ENDPOINTto a suitable trace collecting endpoint in an application likeJaeger(see also theTranslator SRI Jaeger-Demo).Note:the current system Docker (Compose) design only supports OpenTemplate tracing using the internal Jaeger container and may require further refinements to enable use of an external telemetry collector.Running the Web Service within DockerThe Reasoner Validator web service may be run inside a docker container, using Docker Compose.First, from the root project directory, build the local docker containerdocker-composebuildThen, run the service:docker-composeup-dOnce again, go tohttp://localhost/docsto see the service documentation.To stop the service:docker-composedownOf course, the above docker-compose commands may be customized by the user to suit their needs. Note that the docker implementation assumes the use of uvicornChange LogSummary of earlier releases and current Change Log ishere.Code Limitations (implied Future Work?)The release of the reasoner-validator after v2.2.0 will not likely be able to (reliably, if at all) validate TRAPI JSON data models prior to 1.3.0Biolink Model release <= 2.4.8 versus 3.0.0 validation: the reasoner-validator uses the Biolink Model Toolkit. As it happens, the toolkit is not backwards compatible with at least one Biolink Model structural change from release 2.#.# to 3.#.#: the tagging of 'canonical' predicates. That is, the 0.8.10++ toolkit reports canonical <= 2.4.8 model predicates as 'non-canonical'.This release of the Reasoner Validator Web Service will detect TRAPI 1.0.* releases but doesn't strive to be completely backwards compatible with them (considering that TRAPI 1.0.* is totally irrelevant now).The web service validation doesn't do deep validation of the Results part of a TRAPI MessageThe validation is only run on the first 1000 nodes and 100 edges of graphs, to keep the validation time tractable (this risks not having complete coverage of the graph)Biolink Model toolkit is not (yet) cached so changing the model version during use will result in some latency in resultsThe validator service doesn't (yet) deeply validate non-core node and edge slot contents of Message Knowledge GraphsThe validator service doesn't (yet) attempt validation of Query Graph nodes and edges 'constraints' (e.g.biolink:Associationetc.domainandrangeconstraints)Query Graph node 'ids' are not validated except when an associated 'categories' parameter is provided for the given node. In general,Query Graph Validationcould be elaborated.The system should leverage theReasoner Pydantic ModelsCore ContributorsKudos to Patrick Wang, who created the original implementation of the Reasoner-Validator project while with CoVar (an entrepreneurial team contributing to the Biomedical Data Translator).Thanks to Kenneth Morton (CoVar) for his reviews of the latest code.The project is currently being extended and maintained by Richard Bruskiewich (Delphinai Corporation, on the SRI team contributing to Translator)