Package: oca-ds-validator
OCA Data Set ValidatorThis is a Python package for validatingOverlays Capture Architecture (OCA)data sets. It includes three classes:OCADataSet,OCADataSetErr, andOCABundle. For more information about OCA, please checkOCA Specification v1.0.0.OCADataSetrepresents an OCA data set to be validated, and can be loaded from a pandas DataFrame, an OCA Excel Data Entry File, or a CSV file.OCADataSetErrrepresents the result set of an OCA data set validation. This class is generated by the data set validation, contains all the error information, and also provides three methods for a quick overview:overview(),first_error_col(), andget_error_col(attr_name).OCABundlerepresents schema overlays from a loaded.jsonOCA bundle used to validate the data set.DependenciespandaspathlibUsageInstallationInstall the package by typingpip install oca_ds_validatorto the console. Then you could import the classes from any Python scripts.The package can be found here:oca_ds_validatorValidation stepsImport the OCA Bundle usingOCABundle(path).Import the OCA Data Set usingOCADataSet(pandas_dataframe)orOCADataSet.from_path(path).Generate the validation result usingvalidate()method for classOCABundle.fromoca_ds_validatorimportOCADataSet,OCADataSetErr,OCABundletest_bundle=OCABundle("/path/to/oca/bundle.json")test_data=OCADataSet(data_set_dataframe)# test_data = OCADataSet.from_path("/path/to/oca/data_entry_file.xlsx")# test_data = OCADataSet.from_path("/path/to/oca/data_set_file.csv")test_rslt=test_bundle.validate(test_data)########################################################################################## Example of a possible test_rslt:#   attr_err:#     [('missing_attribute',#       'Missing attribute (attribute not found in the data set).'),#      ('unmatched_attribute',#       'Unmatched attribute (attribute not found in the OCA Bundle).')]#   format_err:#     {'attribute_with_format_error_on_row_0': {0: 'Format mismatch.'},#      'array_attribute_without_array_data_on_row_0': {0: 'Valid array required.'},#      'attribute_with_format_error_on_row_42': {42: 'Format mismatch.'},#      'attribute_with_errors_on_row_0_and_1': {0: 'Format mismatch.',#                                               1: 'Valid array required.'},#      'mandatory_attribute_with_missing_data': {0: 'Missing mandatory attribute.'},#      'attribute_without_error': {}}#   ecode_err:  # Not matching any of the entry codes#     {'attribute_with_entry_codes': {0: 'One of the entry codes required.'}}#########################################################################################Optional MessagesThere are three optional boolean arguments to control the message printed.argumentdefault valueusageshow_data_previewFalseIf enabled, prints a pandas preview of the data set before validation.enable_flagged_alarmTrueIf enabled, prints a warning message for the existence of flagged attributes.enable_version_alarmTrueIf enabled, prints a warning message for each overlay that contains an OCA version number different from the development version of this script (1.0).Result ObservationThe errors of the data set is stored in the generatedOCADataSetErrclass.# Prints a brief summary of errors.test_rslt.overview()########################################################################################## Attribute error.# {'missing_attribute'} found in the OCA Bundle but not in the data set;# {'unmatched_attribute'} found in the data set but not in the OCA Bundle.# Found 3 problematic row(s) in the following attribute(s):# {'attribute_with_format_error_on_row_0',#  'array_attribute_without_array_data_on_row_0',#  'attribute_with_format_error_on_row_42',#  'attribute_with_errors_on_row_0_and_1',#  'mandatory_attribute_with_missing_data',#  'attribute_with_entry_codes'}########################################################################################## Prints the information of the first problematic column.test_rslt.first_err_col()########################################################################################## The first problematic column is: attribute_with_format_error_on_row_0# Format error(s) would occur in the following rows:# row 0 : Format mismatch.# No entry code error found in the column.########################################################################################## Prints the information of some certain column.test_rslt.get_err_col("attribute_with_format_error_on_row_42")########################################################################################## Format error(s) would occur in the following rows of column# attribute_with_format_error_on_row_42:# row 42 : Format mismatch.# No entry code error found in the column.#########################################################################################Further Processing# Get objects of full error details.# You may find it useful for data visualization or further analysis.test_rslt.get_attr_err()test_rslt.get_format_err()test_rslt.get_ecode_err()test_rslt.get_char_encode_err()Development StatusThis script is created with support byAgri-food Data Canada, funded byCFREFthrough theFood from Thought grantheld at theUniversity of Guelph. Currently, we do not provide any warranty of any kind regarding the accuracy, security, completeness or reliability of this script or any of its parts.At the moment, this script is developed for the validation of the followingOCA attribute types:Text (with regular expressions)DateTime (with ISO 8601 formats)Array[Type]; for any Types that are not mentioned above, only the validness of the array will be checked.Also, besides the format overlay, the data set will be validated with the following overlays:Conformance Overlay, for any missing mandatory dataEntry Code Overlay, for any data that mismatches entry codesCharacter Encoding, for any data that mismatches the specified character encodingJSON data types areNOTvalidated due to the type coercion while importing Excel or CSV files. We also recommend that you import the data set as Pandas DataFrame to prevent unexpected DateTime formatting by software such as Microsoft Excel.Any validation errors other than the above areNOTguaranteed to be filtered by this script. Please feel free to contact us with any suggestions for future development.You could also find a well-developedOCA ValidatorbyThe Human Colossus Lab(Rust required).LicenseEUPL (European Union Public License), version 1.2We have distilled the most crucial license specifics to make your adoption seamless:see here for details.