Package: cdisc-rules-engine
Supported python versionscdisc-rules-engineOpen source offering of the cdisc rules engineQuick startTo quickly get up and running with CORE, users can download the latest executable version of the engine for their operating system from here:https://github.com/cdisc-org/cdisc-rules-engine/releasesOnce downloaded, simply unzip the file and run the following command based on your Operating System:Windows:.\core.exe validate -s <standard> -v <standard_version> -d path/to/datasets

# ex: .\core.exe validate -s sdtmig -v 3-4 -d .\xpt\Linux/Mac:./core validate -s <standard> -v <standard_version> -d path/to/datasets

# ex: ./core validate -s sdtmig -v 3-4 -d .\xpt\Code formatterThis project uses theblackcode formatter,flake8linter for python andprettierfor JSON, YAML and MD.
It also usespre-committo runblack,flake8andprettierwhen you commit.
Both dependencies are added torequirements.txt.RequiredSetting uppre-commitrequires one extra step. After installing it you have to runpre-commit installThis installspre-commitin your.git/hooksdirectory.Installing dependenciesThese steps should be run before running any tests or core commands using the non compiled version.Create a virtual environment:python -m venv <virtual_environment_name>Activate the virtual environment:./<virtual_environment_name>/bin/activate-- on linux/mac.\<virtual_environment_name>\Scripts\Activate-- on windowsInstall the requirements.python -m pip install -r requirements.txt# From the root directoryRunning The TestsFrom the root of the project run the following command:python -m pytest tests/unit/Running a validationFrom the command lineClone the repository and runpython core.py --helpto see the full list of commands.Runpython core.py validate --helpto see the list of validation options.-ca, --cache TEXT               Relative path to cache files containing pre
                                  loaded metadata and rules
  -ps, --pool-size INTEGER         Number of parallel processes for validation
  -d, --data TEXT                 Path to directory containing data files
  -dp, --dataset-path TEXT        Absolute path to dataset file. Can be specified multiple times.
  -l, --log-level [info|debug|error|critical|disabled|warn]
                                  Sets log level for engine logs, logs are
                                  disabled by default
  -rt, --report-template TEXT     File path of report template to use for
                                  excel output
  -s, --standard TEXT             CDISC standard to validate against
                                  [required]
  -v, --version TEXT              Standard version to validate against
                                  [required]
  -ct, --controlled-terminology-package TEXT
                                  Controlled terminology package to validate
                                  against, can provide more than one
  -o, --output TEXT               Report output file destination
  -of, --output-format [JSON|XLSX]
                                  Output file format
  -rr, --raw-report               Report in a raw format as it is generated by
                                  the engine. This flag must be used only with
                                  --output-format JSON.
  -dv, --define-version TEXT      Define-XML version used for validation
  --whodrug TEXT                  Path to directory with WHODrug dictionary
                                  files
  --meddra TEXT                   Path to directory with MedDRA dictionary
                                  files
  -r, --rules TEXT                Rule core id. ex: CORE-000001. Can be specified multiple times.
  -vo, --verbose-output           Specify this option to print rules as they
                                  are completed
  -p, --progress [verbose_output|disabled|percents|bar]
                                  Defines how to display the validation
                                  progress. By default a progress bar like
                                  "[████████████████████████████--------]
                                  78%"is printed.
  --help                          Show this message and exit.Available log levelsdebug- Display all logsinfo- Display info, warnings, and error logswarn- Display warnings and errorserror- Display only error logscritical- Display critical logsValidate folderTo validate a folder using rules for SDTM-IG version 3.4 use the following command:python core.py validate -s sdtmig -v 3-4 -d path/to/datasetsUnderstanding the Rules ReportThe rules report tab displays the run status of each rule selected for validationThe possible rule run statuses are:SUCCESS- The rule ran and data was validated against the rule. May or may not produce resultsSKIPPED- The rule was unable to be run. Usually due to missing required data, but could also be cause by rule execution errors.Additional Core Commandsupdate-cache - update locally stored cache data (Requires an environment variable -CDISC_LIBRARY_API_KEY)python core.py update-cacheTo obtain an api key, please follow the instructions found here:https://wiki.cdisc.org/display/LIBSUPRT/Getting+Started%3A+Access+to+CDISC+Library+API+using+API+Key+Authentication. Please note it can take up to an hour after sign up to have an api key issuedlist-rules - list rules available in the cachelist all rules:python core.py list-ruleslist rules for standard:python core.py list-rules -s sdtmig -v 3-4list-rule-sets - lists all standards and versions for which rules are available:python core.py list-rule-setstest - Test authored rule given dataset in json format-ca, --cache TEXT               Relative path to cache files containing pre
                                  loaded metadata and rules
  -dp, --dataset-path TEXT        Absolute path to dataset file
  -s, --standard TEXT             CDISC standard to validate against
                                  [required]
  -v, --version TEXT              Standard version to validate against
                                  [required]
  -ct, --controlled-terminology-package TEXT
                                  Controlled terminology package to validate
                                  against, can provide more than one
  -dv, --define-version TEXT      Define-XML version used for validation
  --whodrug TEXT                  Path to directory with WHODrug dictionary
                                  files
  --meddra TEXT                   Path to directory with MedDRA dictionary
                                  files
  -r, --rule TEXT                 Path to rule json file.
  --help                          Show this message and exit.EX:python core.py test -s sdtmig -v 3-4 -dp <path to dataset json file> -r <path to rule json file> --meddra ./meddra/ --whodrug ./whodrug/Note: JSON dataset should match the format provided by the rule editor:{
    "datasets": [{
      "filename": "cm.xpt",
      "label": "Concomitant/Concurrent medications",
      "domain": "CM",
      "variables": [
        {
          "name": "STUDYID",
          "label": "Study Identifier",
          "type": "Char",
          "length": 10
        }
      ],
      "records": {
        "STUDYID": [
          "CDISC-TEST",
          "CDISC-TEST",
          "CDISC-TEST",
          "CDISC-TEST"
        ],
      }
    }
  ]
}list-ct - list ct packages available in the cacheUsage: python core.py list-ct [OPTIONS]

  Command to list the ct packages available in the cache.

Options:
  -c, --cache_path TEXT  Relative path to cache files containing pre loaded
                         metadata and rules
  -s, --subsets TEXT     CT package subset type. Ex: sdtmct. Multiple values
                         allowed
  --help                 Show this message and exit.PyPI Quickstart: Validate data within pythonAn alternative to running the validation from the command line is to instead import the rules engine library in python and run rules against data directly (without needing your data to be in.xptformat).Step 0: Install the librarypip install cdisc-rules-engineIn addition to installing the library, you'll also want to download the rules cache (found in theresources/cachefolder of this repository) and store them somewhere in your project.Step 1: Load the RulesThe rules can be loaded into an in-memory cache by doing the following:importosimportpathlibfrommultiprocessing.managersimportSyncManagerfromcdisc_rules_engine.services.cacheimportInMemoryCacheServiceclassCacheManager(SyncManager):pass# If you're working from a terminal you may need to# use SyncManager directly rather than define CacheManagerCacheManager.register("InMemoryCacheService",InMemoryCacheService)defload_rules_cache(path_to_rules_cache):cache_path=pathlib.Path(path_to_rules_cache)manager=CacheManager()manager.start()cache=manager.InMemoryCacheService()files=next(os.walk(cache_path),(None,None,[]))[2]forfnameinfiles:withopen(cache_path/fname,"rb")asf:cache.add_all(pickle.load(f))returncacheRules in this cache can be accessed by standard and version using theget_rules_cache_keyfunction.fromcdisc_rules_engine.utilities.utilsimportget_rules_cache_keycache=load_rules_cache()# Note that the standard version is separated by a dash, not a periodcache_key_prefix=get_rules_cache_key("sdtmig","3-4")rules=cache.get_all_by_prefix(cache_key_prefix)ruleswill now be a list of dictionaries the following keyscore_ide.g. "CORE-000252"domainse.g.{'Include': ['DM'], 'Exclude': []}or{'Include': ['ALL']}authorreferencesensitivityexecutabilitydescriptionauthoritiesstandardsclassesrule_typeconditionsactionsdatasetsoutput_variablesStep 2: Prepare your dataIn order to pass your data through the rules engine, it must be a pandas dataframe of an SDTM dataset. For example:>>> data
STUDYID DOMAIN USUBJID  AESEQ AESER    AETERM    ... AESDTH AESLIFE AESHOSP
0          AE      001     0     Y     Headache  ...     N       N       N

[1 rows x 19 columns]Before passing this into the rules engine, we need to wrap it in a DatasetVariable.fromcdisc_rules_engine.models.dataset_variableimportDatasetVariabledataset=DatasetVariable(data)Step 3: Run the (relevant) rulesNext, we need to actually run the rules. We can select which rules we want to run based on the domain of the data we're checking and the"Include"and"Exclude"domains of the rule.# Get the rules for the domain AE# (Note: we're ignoring ALL domain rules here)ae_rules=[ruleforruleinrulesif"AE"inrule["domains"].get("Include",[])]There's one last thing we need before we can actually run the rule, and that's aCOREActionsobject. This object will handle generating error messages should the rule fail.To instantiate aCOREActionsobject, we need to pass in the following:results: An array to which errors will be appendedvariable: Our DatasetVariabledomain: e.g. "AE"rule: Our rulefromcdisc_rules_engine.models.actionsimportCOREActionsrule=ae_rules[0]results=[]core_actions=COREActions(results,variable=dataset,domain="AE",rule=rule)All that's left is to run the rule!frombusiness_rules.engineimportrunwas_triggered=run(rule=rule,defined_variables=dataset_variable,defined_actions=core_actions,)Step 5: Interpret the resultsThe return value of run will tell us if the rule was triggered.AFalsevalue means that there were no errorsATruevalue means that there were errorsIf there were errors, they will have been appended to the results array passed into yourCOREActionsinstance. Here's an example error:{'executionStatus':'success','domain':'AE','variables':['AESLIFE'],'message':'AESLIFE is completed, but not equal to "N" or "Y"','errors':[{'value':{'AESLIFE':'Maybe'},'row':1}]}Creating an executable versionLinuxpyinstaller core.py --add-data=venv/lib/python3.9/site-packages/xmlschema/schemas:xmlschema/schemas --add-data=resources/cache:resources/cache --add-data=resources/templates:resources/templatesWindowspyinstaller core.py --add-data=".venv/Lib/site-packages/xmlschema/schemas;xmlschema/schemas" --add-data="resources/cache;resources/cache" --add-data="resources/templates;resources/templates"Note .venv should be replaced with path to python installation or virtual environmentThis will create an executable version in thedistfolder. The version does not require having Python installed and
can be launched by runningcorescript with all necessary CLI arguments.Creating .whl fileAll non-python files should be listed inMANIFEST.into be included in the distribution.
Files must be in python package.Unix/MacOSpython3 -m pip install --upgrade buildpython3 -m buildTo install from dist folderpip3 install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whlTo upload built distributive to pypipython3 -m pip install --upgrade twinepython3 -m twine upload --repository {repository_name} dist/*Windows(Untested)py -m pip install --upgrade buildpy -m buildTo install from dist folderpip install {path_to_file}/cdisc_rules_engine-{version}-py3-none-any.whlTo upload built distributive to pypipy -m pip install --upgrade twinepy -m twine upload --repository {repository_name} dist/*