Package: google-pso-data-validator
Data Validation ToolThe Data Validation Tool is an open sourced Python CLI tool based on theIbis frameworkthat compares heterogeneous data source tables with multi-leveled validation
functions.Data validation is a critical step in a data warehouse, database, or data lake
migration project where data from both the source and the target tables are
compared to ensure they are matched and correct after each migration step
(e.g. data and schema migration, SQL script translation, ETL migration, etc.).
The Data Validation Tool (DVT) provides an automated and repeatable solution to
perform this task.DVT supports the following validations:Column validation (count, sum, avg, min, max, stddev, group by)Row validation (Not supported for FileSystem connections)Schema validationCustom Query validationAd hoc SQL explorationDVT supports the following connection types:AlloyDBBigQueryDB2FileSystemHiveImpalaMSSQLMySQLOraclePostgresRedshiftSpannerTeradataSnowflakeTheConnectionspage provides details about how to create
and list connections for the validation tool.DisclaimerThis is not an officially supported Google product. Please be aware that bugs may lurk, and that we reserve the right to make small backwards-incompatible changes. Feel free to open bugs or feature requests, or contribute directly
(seeCONTRIBUTING.mdfor details).InstallationTheInstallationpage describes the prerequisites and
setup steps needed to install and use the Data Validation Tool.UsageBefore using this tool, you will need to create connections to the source and
target tables. Once the connections are created, you can run validations on
those tables. Validation results can be printed to stdout (default) or outputted
to BigQuery (recommended). DVT also allows you to save and edit validation
configurations in a YAML or JSON file. This is useful for running common validations or
updating the configuration.Managing ConnectionsBefore running validations, DVT requires setting up a source and target connection.
These connections can be stored locally or in a GCS directory. To create connections,
please review theConnectionspage.Running ValidationsThe CLI is the main interface to use this tool and it has several different
commands which can be used to create and run validations. Below are the command
syntax and options for running validations.Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
(Examples Here). See theValidation Logicsection
to learn more about how DVT uses the CLI to generate SQL queries.Note that we do not support nested or complex columns for column or row validations.Column ValidationsBelow is the command syntax for column validations. To run a grouped column
validation, simply specify the--grouped-columnsflag.You can specify a list of string columns for aggregations in order to calculate
an aggregation over thelength(string_col). Similarly, you can specify timestamp/date
columns for aggregation over theunix_seconds(timestamp_col). Running an aggregation
over all columns ('*') will only run over numeric columns, unless the--wildcard-include-string-lenor--wildcard-include-timestampflags are present.data-validation (--verbose or -v) (--log-level or -ll) validate column
  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
                        Comma separated list of tables in the form schema.table=target_schema.target_table
                        Target schema name and table name are optional.
                        i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
  [--grouped-columns or -gc GROUPED_COLUMNS]
                        Comma separated list of columns for Group By i.e col_a,col_b
  [--count COLUMNS]     Comma separated list of columns for count or * for all columns
  [--sum COLUMNS]       Comma separated list of columns for sum or * for all numeric
  [--min COLUMNS]       Comma separated list of columns for min or * for all numeric
  [--max COLUMNS]       Comma separated list of columns for max or * for all numeric
  [--avg COLUMNS]       Comma separated list of columns for avg or * for all numeric
  [--std COLUMNS]       Comma separated list of columns for stddev_samp or * for all numeric
  [--exclude-columns or -ec]   Flag to indicate the list of columns provided should be excluded and not included.
  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
                        BigQuery destination for validation results. Defaults to stdout.
                        See: *Validation Reports* section
  [--service-account or -sa PATH_TO_SA_KEY]
                        Service account to use for BigQuery result handler output.
  [--wildcard-include-string-len or -wis]
                        If flag is present, include string columns in aggregation as len(string_col)
  [--wildcard-include-timestamp or -wit]
                        If flag is present, include timestamp/date columns in aggregation as unix_seconds(ts_col)
  [--cast-to-bigint or -ctb]
                        If flag is present, cast all int32 columns to int64 before aggregation
  [--filters SOURCE_FILTER:TARGET_FILTER]
                        Colon separated string values of source and target filters.
                        If target filter is not provided, the source filter will run on source and target tables.
                        See: *Filters* section
  [--config-file or -c CONFIG_FILE]
                        YAML Config File Path to be used for storing validations and other features.
                        See: *Running DVT with YAML Configuration Files* section
  [--config-file-json or -cj CONFIG_FILE_JSON]
                        JSON Config File Path to be used for storing validations only for application purposes.
  [--threshold or -th THRESHOLD]
                        Float value. Maximum pct_difference allowed for validation to be considered a success. Defaults to 0.0
  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
                        Comma-separated key value pair labels for the run.
  [--format or -fmt]    Format for stdout output. Supported formats are (text, csv, json, table).
                        Defaults to table.
  [--filter-status or -fs STATUSES_LIST]
                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.The default aggregation type is a 'COUNT *', which will run in addition to the validations you specify. To remove this default,
useYAML configs.TheExamplespage provides many examples of how a tool can be used to run powerful validations without writing any queries.Row Validations(Note: Row hash validation not supported for FileSystem connections.
In addition, please note that SHA256 is not a supported function on Teradata systems.
If you wish to perform this comparison on Teradata you will need todeploy a UDF to perform the conversion.)Below is the command syntax for row validations. In order to run row level
validations you need to pass a--primary-keyflag which defines what field(s)
the validation will be compared on, as well as either the--comparison-fieldsflag
or the--hashflag. SeePrimary KeyssectionThe--comparison-fieldsflag specifies the values (e.g. columns) whose raw values will be compared
based on the primary key join. The--hashflag will run a checksum across specified columns in
the table. This will include casting to string, sanitizing the data (ifnull, rtrim, upper), concatenating,
and finally hashing the row.Under the hood, row validation usesCalculated Fieldsto
apply functions such as IFNULL() or RTRIM(). These can be edited in the YAML or JSON config file to customize your row validation.data-validation (--verbose or -v) (--log-level or -ll) validate row
  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
                        Comma separated list of tables in the form schema.table=target_schema.target_table
                        Target schema name and table name are optional.
                        i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
  --primary-keys or -pk PRIMARY_KEYS
                        Comma separated list of columns to use as primary keys.  See *Primary Keys* section
  --comparison-fields or -comp-fields FIELDS
                        Comma separated list of columns to compare. Can either be a physical column or an alias
                        See: *Calculated Fields* section for details
  --hash COLUMNS        Comma separated list of columns to hash or * for all columns
  --concat COLUMNS      Comma separated list of columns to concatenate or * for all columns (use if a common hash function is not available between databases)
  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
                        BigQuery destination for validation results. Defaults to stdout.
                        See: *Validation Reports* section
  [--service-account or -sa PATH_TO_SA_KEY]
                        Service account to use for BigQuery result handler output.
  [--filters SOURCE_FILTER:TARGET_FILTER]
                        Colon separated string values of source and target filters.
                        If target filter is not provided, the source filter will run on source and target tables.
                        See: *Filters* section
  [--config-file or -c CONFIG_FILE]
                        YAML Config File Path to be used for storing validations and other features.
                        See: *Running DVT with YAML Configuration Files* section
  [--config-file-json or -cj CONFIG_FILE_JSON]
                        JSON Config File Path to be used for storing validations only for application purposes.
  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
                        Comma-separated key value pair labels for the run.
  [--format or -fmt]    Format for stdout output. Supported formats are (text, csv, json, table).
                        Defaults to table.
  [--use-random-row or -rr]
                        Finds a set of random rows of the first primary key supplied.
  [--random-row-batch-size or -rbs]
                        Row batch size used for random row filters (default 10,000).
  [--filter-status or -fs STATUSES_LIST]
                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.Generate Table Partitions for Large Table Row ValidationsWhen performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. SeePrimary KeyssectionThe command generates and stores multiple YAML configs that represent chunks of the large table using filters (WHERE primary_key(s) >= X AND primary_key(s) < Y). You can then run the configs in the directory serially (or in parallel in multiple containers, VMs) with thedata-validation configs run --config-dir PATHcommand as describedhere.The command takes the same parameters as required forRow Validationplusa few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions,partition-keyis no longer supported.data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions

  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
                        Comma separated list of tables in the form schema.table=target_schema.target_table
                        Target schema name and table name are optional.
                        i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
  --primary-keys PRIMARY_KEYS, -pk PRIMARY_KEYS
                        Comma separated list of primary key columns 'col_a,col_b'.  See *Primary Keys* section
  --comparison-fields or -comp-fields FIELDS
                        Comma separated list of columns to compare. Can either be a physical column or an alias
                        See: *Calculated Fields* section for details
  --hash COLUMNS        Comma separated list of columns to hash or * for all columns
  --concat COLUMNS      Comma separated list of columns to concatenate or * for all columns (use if a common hash function is not available between databases)
  --config-dir CONFIG_DIR, -cdir CONFIG_DIR
                        Directory Path to store YAML Config Files
                        GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                        Local: Provide a relative path of the target directory. Eg: `partitions_dir`
  --partition-num [1-1000], -pn [1-1000]
                        Number of partitions/config files to generate
                        In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
  [--filters SOURCE_FILTER:TARGET_FILTER]
                        Colon separated string values of source and target filters.
                        If target filter is not provided, the source filter will run on source and target tables.
                        See: *Filters* sectionSchema ValidationsBelow is the syntax for schema validations. These can be used to compare case insensitive column names and
types between source and target.Note: An exclamation point before a data type (!string) signifies the column is non-nullable or required.data-validation (--verbose or -v) (--log-level or -ll) validate schema
  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
                        Comma separated list of tables in the form schema.table=target_schema.target_table
                        Target schema name and table name are optional.
                        e.g.: 'bigquery-public-data.new_york_citibike.citibike_trips'
  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
                        BigQuery destination for validation results. Defaults to stdout.
                        See: *Validation Reports* section
  [--service-account or -sa PATH_TO_SA_KEY]
                        Service account to use for BigQuery result handler output.
  [--config-file or -c CONFIG_FILE]
                        YAML Config File Path to be used for storing validations and other features.
                        See: *Running DVT with YAML Configuration Files* section
  [--config-file-json or -cj CONFIG_FILE_JSON]
                        JSON Config File Path to be used for storing validations only for application purposes.
  [--format or -fmt]    Format for stdout output. Supported formats are (text, csv, json, table).
                        Defaults  to table.
  [--filter-status or -fs STATUSES_LIST]
                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail).
                        If no list is provided, all statuses are returned.
  [--exclusion-columns or -ec EXCLUSION_COLUMNS]
                        Comma separated list of columns to be excluded from the schema validation, e.g.: col_a,col_b.
  [--allow-list or -al ALLOW_LIST]
                        Comma separated list of data-type mappings of source and destination data sources which will be validated in case of missing data types in destination data source. e.g: "decimal(4,2):decimal(5,4),!string:string"
  [--allow-list-file ALLOW_LIST_FILE, -alf ALLOW_LIST_FILE]
                        YAML file containing default --allow-list mappings. Can be used in conjunction with --allow-list.
                        e.g.: samples/allow_list/oracle_to_bigquery.yaml or gs://dvt-allow-list-files/oracle_to_bigquery.yaml
                        See example files in samples/allow_list/.Custom Query Column ValidationsBelow is the command syntax for custom query column validations.data-validation (--verbose or -v) (--log-level or -ll) validate custom-query column
  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
                        Source sql query
                        Either --source-query or --source-query-file must be provided
  --source-query-file  SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
                        File containing the source sql commands
  --target-query TARGET_QUERY, -tq TARGET_QUERY
                        Target sql query
                        Either --target-query or --target-query-file must be provided
  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
                        File containing the target sql commands
  [--count COLUMNS]     Comma separated list of columns for count or * for all columns
  [--sum COLUMNS]       Comma separated list of columns for sum or * for all numeric
  [--min COLUMNS]       Comma separated list of columns for min or * for all numeric
  [--max COLUMNS]       Comma separated list of columns for max or * for all numeric
  [--avg COLUMNS]       Comma separated list of columns for avg or * for all numeric
  [--std COLUMNS]       Comma separated list of columns for stddev_samp or * for all numeric
  [--exclude-columns or -ec]   Flag to indicate the list of columns provided should be excluded and not included.
  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
                        BigQuery destination for validation results. Defaults to stdout.
                        See: *Validation Reports* section
  [--service-account or -sa PATH_TO_SA_KEY]
                        Service account to use for BigQuery result handler output.
  [--config-file or -c CONFIG_FILE]
                        YAML Config File Path to be used for storing validations and other features.
                        See: *Running DVT with YAML Configuration Files* section
  [--config-file-json or -cj CONFIG_FILE_JSON]
                        JSON Config File Path to be used for storing validations only for application purposes.
  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
                        Comma-separated key value pair labels for the run.
  [--format or -fmt]    Format for stdout output. Supported formats are (text, csv, json, table).
                        Defaults to table.
  [--filter-status or -fs STATUSES_LIST]
                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.The default aggregation type is a 'COUNT *'. If no aggregation flag (i.e count,
sum , min, etc.) is provided, the default aggregation will run.TheExamplespage provides few examples of how this tool can be used to run custom query validations.Custom Query Row Validations(Note: Custom query row validation is not supported for FileSystem connections. Struct and array data types are not currently supported.)Below is the command syntax for row validations. In order to run row level
validations you need to pass--hashflag which will specify the fields
of the custom query result that will be concatenated and hashed. The primary key should be included
in the SELECT statement of both source_query.sql and target_query.sql.  SeePrimary KeyssectionBelow is the command syntax for custom query row validations.data-validation (--verbose or -v) (--log-level or -ll) validate custom-query row
  --source-conn or -sc SOURCE_CONN
                        Source connection details
                        See: *Data Source Configurations* section for each data source
  --target-conn or -tc TARGET_CONN
                        Target connection details
                        See: *Connections* section for each data source
  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
                        Source sql query
                        Either --source-query or --source-query-file must be provided
  --source-query-file SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
                        File containing the source sql commands
  --target-query TARGET_QUERY, -tq TARGET_QUERY
                        Target sql query
                        Either --target-query or --target-query-file must be provided
  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
                        File containing the target sql commands
  --comparison-fields or -comp-fields FIELDS
                        Comma separated list of columns to compare. Can either be a physical column or an alias
                        See: *Calculated Fields* section for details
  --hash '*'            '*' to hash all columns.
  --concat COLUMNS      Comma separated list of columns to concatenate or * for all columns
                        (use if a common hash function is not available between databases)
  --primary-key or -pk JOIN_KEY
                       Common column between source and target tables for join
  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
                        BigQuery destination for validation results. Defaults to stdout.
                        See: *Validation Reports* section
  [--service-account or -sa PATH_TO_SA_KEY]
                        Service account to use for BigQuery result handler output.
  [--config-file or -c CONFIG_FILE]
                        YAML Config File Path to be used for storing validations and other features.
                        See: *Running DVT with YAML Configuration Files* section
  [--config-file-json or -cj CONFIG_FILE_JSON]
                        JSON Config File Path to be used for storing validations only for application purposes.
  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
                        Comma-separated key value pair labels for the run.
  [--format or -fmt]    Format for stdout output. Supported formats are (text, csv, json, table).
                        Defaults to table.
  [--filter-status or -fs STATUSES_LIST]
                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.TheExamplespage provides few examples of how this tool can be used to run custom query row validations.Dry Run ValidationThevalidatecommand takes a--dry-runcommand line flag that prints source
and target SQL to stdout as JSON in lieu of performing a validation:data-validation (--verbose or -v) (--log-level or -ll) validate
  [--dry-run or -dr]    Prints source and target SQL to stdout in lieu of performing a validation.For example, this flag can be used as follows:>data-validationvalidate--dry-runrow\-scmy_bq_conn\-tcmy_bq_conn\-tblsbigquery-public-data.new_york_citibike.citibike_stations\--primary-keysstation_id\--hash'*'{"source_query":"SELECT `hash__all`, `station_id`\nFROM ...","target_query":"SELECT `hash__all`, `station_id`\nFROM ..."}Running DVT with YAML Configuration FilesRunning DVT with YAML configuration files is the recommended approach if:you want to customize the configuration for any given validation ORyou want to run DVT at scale (i.e. row validations across many partitions)
Nearly every validation command can take the argument-c <file-name>(thegenerate-table-partitionscommand takes-cdir <directory-name>) where one or more yaml files are produced. These yaml files can be modified for custom validations.The following command creates a YAML file for the validation of thenew_york_citibiketable:data-validation validate column -sc my_bq_conn -tc my_bq_conn -tbls bigquery-public-data.new_york_citibike.citibike_trips -c citibike.yaml.The validation config file is saved to the GCS path specified by thePSO_DV_CONFIG_HOMEenv variable if that has been set; otherwise, it is saved to wherever the tool is run.You can now edit the YAML file if, for example, thenew_york_citibiketable is
stored in datasets that have different names in the source and target systems.
Once the file is updated and saved, the following command runs the
validation:data-validation (--verbose or -v) (--log-level or -ll) configs run -c citibike.yamlYou can also specify a directory to read from using the --config-dir (or -cdir) flag. All yaml files in that directory will be run as validations. Support both local and GCS paths. For example:data-validation configs run -cdir ./my-validations/

OR

data-validation configs run -cdir gs://my-bucket/my-validations/View the complete YAML file for a Grouped Column validation on theExamplespage.You can view a list of all saved validation YAML files usingdata-validation configs list, and print a YAML config usingdata-validation configs get -c citibike.yaml.Scaling DVT to run 10's to 1000's of validations concurrentlyThe above exampleconfigs run -cdirshows how you can run multiple validations with one command. If the directory used has been created bygenerate-table-partitions, you will have partition files numbered from0000.yamlto<partno-1>.yaml, where<partno>is the total number of partitions. These can be run as mentioned above and the partitions will be validated in order. This could take a long time if the number of partitions is large.DVT validations can be run concurrently (horizontal scaling) using GKE (Kubernetes Jobs) or Cloud Run Jobs. In order to run DVT in a container, you have to build a docker image,see instructions. Set thePSO_DV_CONFIG_HOMEenvironment variable to point to a GCS prefix where the connection configuration files are stored. In Cloud Run you can use the option--set-env-varsor--update-env-varsto passthe environment variable. We recommend that you use thebq-result-handlerto save your validation results. In order to validate partitions concurrently, run DVT in Kubernetes or Cloud Run as shown below:data-validation (--verbose or -v) (--log-level or -ll) configs run
  [--kube-completions or -kc] Specifies that validation is being run in Kubernetes or Cloud Run in indexed completion mode.
              specifies to DVT that validation being run in indexed completion mode in Kubernetes or as multiple independent tasks in Cloud Run.
  --config-dir or -cdir GCS_DIRECTORY 
              where GCS_DIRECTORY = CONFIG_DIR/SOURCE_SCHEMA.SOURCE_TABLE, where CONFIG_DIR, SOURCE_SCHEMA and SOURCE_TABLE are
              the arguments provided to generate-table-partitions to generate the partition yamls. GCS_DIRECTORY is the directory
              where the partition files numbered 0000.yaml to <partno-1>.yaml are stored.If you are not familiar with Kubernetes or Cloud Run Jobs, review theinternal docsfor an overview of the orchestration of multiple validations.The Cloud Run and Kubernetes pods must run in a network with access to the database servers. Every Cloud Run job is associated with aservice account. You need to ensure that this service account has access to Google Cloud Storage (to read connection configuration and yaml files) and BigQuery (to publish results). If you are using Kubernetes, you will need to use a service account with the same privileges as mentioned for Cloud Run. In Kubernetes, you will need to set upworkload identityso the DVT container can impersonate the service account.In Cloud Run, thejobmust be run as multiple, independent tasks with the task count set to the number of partitions generated. In Kubernetes, set the number of completions to the number of partitions generated - seeKubernetes Parallel Jobs. The option--kube-completions or -kctells DVT that many DVT containers are running in a Kubernetes cluster. Each DVT container only validates the specific partition yaml (based on the index assigned by Kubernetes control plane). If the-kcoption is used and you are not running in indexed mode, you will receive a warning and the container will process all the validations sequentially. If the-kcoption is used and a config directory is not provided (a--config-fileis provided instead), a warning is issued.By default, each partition validation is retried up to 3 times if there is an error. In Kubernetes and Cloud Run, you can set the parallelism to the number you want. Keep in mind that if you are validating 1000's of partitions in parallel, you may find that setting the parallelism too high (say 100) may result in timeouts and slow down the validation.Validation ReportsThe result handlers tell DVT where to store the results of
each validation. The tool can write the results of a validation run to Google
BigQuery or print to stdout (default). View the schema of the results
tablehere.To output to BigQuery, simply include the-bqrhflag during a validation run
like so:data-validation validate column
  -sc bq_conn
  -tc bq_conn
  -tbls bigquery-public-data.new_york_citibike.citibike_trips
  -bqrh project_id.dataset.table
  -sa service-acct@project.iam.gserviceaccount.comAd Hoc SQL ExplorationThere are many occasions where you need to explore a data source while running
validations. To avoid the need to open and install a new client, the CLI allows
you to run ad hoc queries.data-validation query
  --conn or -c CONN
          The connection name to be queried
  --query or -q QUERY
          The raw query to run against the supplied connectionBuilding Matched Table ListsCreating the list of matched tables can be a hassle. We have added a feature
which may help you to match all of the tables together between source and
target. Thefind-tablescommand:Pulls all tables in the source (applying a suppliedallowed-schemasfilter)Pulls all tables from the targetUses Jaro Similarity algorithm to match tablesFinally, it prints a JSON list of tables which can be a reference for the
validation run config.Note that our default value for thescore-cutoffparameter is 1 and it seeks for identical matches. If no matches occur, reduce this value as deemed necessary. By using smaller numbers such as 0.7, 0.65 etc you can get more matches. For reference, we make use ofthis jaro_similarity methodfor the string comparison.data-validation find-tables --source-conn source --target-conn target \
    --allowed-schemas pso_data_validator \
    --score-cutoff 1Using Beta CLI FeaturesThere may be occasions we want to release a new CLI feature under a Beta flag.
Any features under Beta may or may not make their way to production. However, if
there is a Beta feature you wish to use than it can be accessed using the
following.data-validation beta --help[Beta] Deploy Data Validation as a Local ServiceIf you wish to use Data Validation as a Flask service, the following command
will help. This same logic is also expected to be used for Cloud Run, Cloud
Functions, and other deployment services.data-validation beta deployValidation LogicAggregated FieldsAggregate fields contain the SQL fields that you want to produce an aggregate
for. Currently the functionsCOUNT(),AVG(),SUM(),MIN(),MAX(),
andSTDDEV_SAMP()are supported.Here is a sample aggregate config:validations:-aggregates:-field_alias:countsource_column:nulltarget_column:nulltype:count-field_alias:count__tripdurationsource_column:tripdurationtarget_column:tripdurationtype:count-field_alias:sum__tripdurationsource_column:tripdurationtarget_column:tripdurationtype:sumIf you are aggregating columns with large values, you can CAST() before aggregation
with calculated fields as shown inthis example.FiltersFilters let you apply a WHERE statement to your validation query (ie.SELECT * FROM table WHERE created_at > 30 days ago AND region_id = 71;). The filter is
written in the syntax of the given source.Note that you are writing the query to execute, which does not have to match
between source and target as long as the results can be expected to align. If
the target filter is omitted, the source filter will run on both the source and
target tables.Primary KeysIn many cases, validations (e.g. count, min, max etc) produce one row per table. The comparison between the source
and target table is to compare the value for each column in the source with the value of the column in the target.grouped-columnsvalidation andvalidate rowproduce multiple rows per table. Data Validation Tool needs one or more columns to uniquely identify each row so the source and target can be compared. Data Validation Tool refers to these columns as primary keys. These do not need to be primary keys in the table. The only requirement is that the keys uniquely identify the row in the results.Grouped ColumnsGrouped Columns contain the fields you want your aggregations to be broken out
by, e.g.SELECT last_updated::DATE, COUNT(*) FROM my.tablewill produce a
resultset that breaks down the count of rows per calendar date.Hash, Concat, and Comparison FieldsRow level validations can involve either a hash/checksum, concat, or comparison fields.
A hash validation (--hash '*') will first sanitize the data with the following
operations on all or selected columns: CAST to string, IFNULL replace with a default
replacement string, RSTRIP, and UPPER. Then, it will CONCAT() the results
and run a SHA256() hash and compare the source and target results. Since each row will
be returned in the result set, it is recommended to utilize the--use-random-rowfeature
to validate a subset of the table.Please note that SHA256 is not a supported function on Teradata systems. If you wish to perform
this comparison on Teradata you will need todeploy a UDF to perform the conversion.The concat validation (--concat '*') will do everything up until the hash. It will sanitize
and concatenate the specified columns, and then value compare the results.Comparison field validations (--comp-fields column) involve an value comparison of the
column values. These values will be compared via a JOIN on their corresponding primary
key and will be evaluated for an exact match.See hash and comparison field validations in theExamplespage.Calculated FieldsSometimes direct comparisons are not feasible between databases due to
differences in how particular data types may be handled. These differences can
be resolved by applying functions to columns in the query itself.
Examples might include trimming whitespace from a string, converting strings to
a single case to compare case insensitivity, or rounding numeric types to a
significant figure.Once a calculated field is defined, it can be referenced by other calculated
fields at any "depth" or higher. Depth controls how many subqueries are executed
in the resulting query. For example, with the following YAML config:-calculated_fields:-field_alias:rtrim_col_asource_calculated_columns:['col_a']target_calculated_columns:['col_a']type:rtrimdepth:0# generated off of a native column-field_alias:ltrim_col_asource_calculated_columns:['col_b']target_calculated_columns:['col_b']type:ltrimdepth:0# generated off of a native column-field_alias:concat_col_a_col_bsource_calculated_columns:['rtrim_col_a','ltrim_col_b']target_calculated_columns:['rtrim_col_a','ltrim_col_b']type:concatdepth:1# calculated one query aboveis equivalent to the following SQL query:SELECTCONCAT(rtrim_col_a,rtrim_col_b)ASconcat_col_a_col_bFROM(SELECTRTRIM(col_a)ASrtrim_col_a,LTRIM(col_b)ASltrim_col_bFROMmy.table)astable_0If you generate the config file for a row validation, you can see that it uses
calculated fields to generate the query. You can also use calculated fields
in column level validations to generate the length of a string, or cast
a INT field to BIGINT for aggregations.See theExamples pagefor a sample
cast to NUMERIC.Custom Calculated FieldsDVT supports certain functions required for row hash validation natively (i.e. CAST() and CONCAT()),
which are listed in the CalculatedField() class methods in theQueryBuilder.You can also specify custom functions (i.e. replace() or truncate()) from the Ibis expressionAPI reference. Keep in mind these will run
on both source and target systems. You will need to specify the Ibis API expression and the parameters
required, if any, with the 'params' block as shown below:-calculated_fields:-depth:0field_alias:format_start_timesource_calculated_columns:-start_timetarget_calculated_columns:-start_timetype:customibis_expr:ibis.expr.types.TemporalValue.strftimeparams:-format_str:'%m%d%Y'The above block references theTemporalValue.strftimeIbis API expression.
See theExamples pagefor a sample YAML with a custom calculated field.ContributingContributions are welcome. See theContributing guidefor details.