Package: hashmap-data-validator
Hashmap Data ValidatorAboutHashmap Data Validator, or hdv, tool that is used to validate the data in two database tables. HDV uses great expectations to run validation on the two tables. It currently supports connections to Snowflake with the Snowflake-connector-python and to
Oracle via cx_Oracle. HDV currently runs expectations on row count and row hash values to validate the tables.How To UseThe user installs the package via PyPi with:pipinstallhashmap-data-validatorAfter installation, user needs to run a.pyfile with the following import:import hdvTwo.ymlfiles will then be created in the user'shomedirectory with the following path:.hashmap_data_validator/hdv_profiles.ymland.hashmap_data_validator/hdv_configuration.ymlThe user then configures both of the newly created YAML files accordingly. (Examples below)After configuration is complete, the user can call and run validation withhdvfrom the command line with:hdvHDV writes the validation results to a newly createdvalidation_results.htmlfile (if it does not exist) in the directory where the command is called and opens that file in your browserTime To Execute Method2 million total rows = ~15 seconds8 million total rows = ~2 minutes15 million total rows = ~3.5 minutesUser DocumentationClassesConfiguration templates for HDV supported databases. Add these YAML templates to yourhdv_configuration.ymland configure accordingly.OracleOracle database resource.base classOracleResourceconfigurationrequiredenv: section name inhdvprofile yml file for connection informationtable_name: table name to validateoptionalchunk_size: the number of rows in the table to grab in batches (iterates over table with offset)offset: the number of rows to skip between table batch grabsNOTE:Only includechunk_sizeandoffsetvalues if you would like to validate a samples of your tables.oracle:type:OracleResourceconf:env:oracletable_name:<oracle_db>.<oracle_table>chunk_size:100000# batches are grabbed in chunks of 100000 rows (optional)offset:200000# skips every 200000 rows in the table after a batch grab (optional)SnowflakeSnowflake database resource.base classSnowflakeResourceconfigurationrequiredenv: section name inhdvprofile yml file for connection informationtable_name: table name to validateoptionalchunk_size: the number of rows in the table to grab in batches (iterates over table with offset)offset: the number of rows to skip between table batch grabsNOTE:Only includechunk_sizeandoffsetvalues if you would like to validate a samples of your tables.oracle:type:SnowflakeResourceconf:env:snowflaketable_name:<snowflake_table_name>chunk_size:100000# batches are grabbed in chunks of 100000 rows (optional)offset:200000# skips every 200000 rows in the table after a batch grab (optional)More database support to comeHDV Configuration SetupAfter you run the python file withimport hdv, two configuration files will be created in yourhomedirectory under the.hashmap_data_validator/folder.hdv_profiles.ymlis where you configure your connection credentials for the databases you would like to validate. The default
file will look similar to this:dev:oracle:host:<host>port:<port>sid:<sid>user:<oracle_username>password:<oracle_password>client_library_dir:<path to oracle client library files e.g. C:\instantclient-basic-windows.x64-19.9.0.0.0dbru\instantclient_19_9>snowflake:account:<account>role:<role>warehouse:<warehouse_name>database:<database_name>schema:<schema_name>user:<snowflake_username>password:<snowflake_password>HDV uses this file to manage the connections to the necessary databases. HDV only supports Oracle and Snowflake at the moment,
but we plan to extend support to other databases in the future.
In this file, you can add your credentials for any database that HDV supports. When the validation method is called, HDV is pointed to the individual configurations in this file
and connects using the corresponding credentials.Note: you can add any number of database configurations in this file. The above is just a starting template with two databases. For example, you could have multiple Snowflake
configurations. However, the names would need to be unique (e.g.snowflake_1,snowflake_2)hdv_configuration.ymlis where you configure the specifics of your validation. The file should look similar to below:version:1betavalidations:from:oracle_1to:snowflake_1resources:oracle_1:type:OracleResourceconf:env:oracletable_name:<oracle_db>.<oracle_table>chunk_size:<optional>offset:<optional>snowflake_1:type:SnowflakeResourceconf:env:snowflaketable_name:<snowflake_table>chunk_size:<optional>offset:<optional>dummy_database:type:TestResourceconf:env:test_environmenttable_name:<test_table>chunk_size:<optional>offset:<optional>As you can see above, you can add any number of resources in this file. Then, to run a validation, you call the
specific resources in thevalidationssection of the YAML.The example above will run a validation on theoracle_1andsnowflake_1resources, which are both configured in the above file as well. Notice that theenvvalues of theoracle_1andsnowflake_1resources
point to named configurations in the examplehdv_profiles.ymlshown above. Thetypevalues point to which HDV class type to use in validation (SnowflakeResourceandOracleResourcein this case). Refer above to the HDV supported class types.If I wanted to run a validation on different tables, I merely have to alter thetoandfromvalues to point to different resources configured in the file
or I could just change the current configurations.Note: You can configure any number of resources inhdv_configuration.yml