Package: schemaql
SchemaQLA testing and auditing tool inspired by dbt, for those not usingdbt.InstallationFork and clone to repo to a local folder.Create a new Python virtual environment in this folder and activate it.In the local repo folder, runpip install -r  requirementsto install the dependent packages in this virtualenv.In the local repo folder, runpip install -e .to install a dev version locally.You should now be able to runschemaql -hfrom the command line.schemaql-h
usage:schemaql[-h][-pNone][-cconfig.yml][-xconnections.yml][action]positionalarguments:actionAction('test',or'generate')optionalarguments:-h,--helpshowthishelpmessageandexit-pNone,--projectNoneProject-cconfig.yml,--config-fileconfig.ymlConfigfile-xconnections.yml,--connections-fileconnections.ymlConnectionsfileConfigurationSchemaQL needs to*.ymlfor configuration:Inconnections.ymlyou define how to connect to one or more of your data warehouse connections like so:project-1-snowflake:type:snowflakeaccount:<my_account>user:<my_user>password:<my_password>database:<my_database># optional:warehouse:<my_warehouse>role:<my_role>schema:<my_initial_schema>project-2-bigquery:type:bigquerydatabase:<my_bigquery_project>credentials_path:<path_to_my_service_account_credentials.json>supports_multi_insert:FalseInconfig.yml, you define the following:Logging, whereoutputis a directory relative to the project path.logs:output:logsacollectorwhich is a definition for how you want test results to be collected. Currently supported are:jsoncsvdatabaseForjsonandcsv, you only need to provide anoutputpath.collector:type:csvoutput:outputFordatabasecollection, you need to provide the name of a connection (fromconnections.yml) and a destination table viaoutput:collector:type:databaseconnection:project-1-snowflakeoutput:test_resultsThecollectorconnection does not need to the same connection, or even the same connection type as the project connection. So, tests could be run against BigQuery but tests results could be collected using Snowflake or Postgres.projectswhich is a combination of a connection and a list of which databases and schemas you want to work with. If you don't define any schemas within the database key, all schemas will be processed.projects:project-1:connection:project-1-snowflakeschema:database_1:-schema_1-schema_2project-2:connection:project-2-bigqueryschema:my_bg_project_1my_bg_project_2- data_set_1- data_set_2UsageGenerateSchemaQL runs tests against schema information contained inymlfiles. You can either write these from scratch, use your existingdbtschema files, or useschemaqlto generate them.Generates schema files for all projects:schemaqlgenerateGenerates schema files for"my_project"only:schemaqlgenerate-pmy_projectTestschemaqltestTest"my_project"only:schemaqltest-pmy_projectBuilt-in TestsSchema Testsnot_nullChecks if column values areNULLmodels:-name:my_tablecolumns:-name:col_1tests:-not_nullrelationshipsChecks if column values match values from column in other entitymodels:-name:my_tablecolumns:-name:col_1tests:-relationships:to:my_other_tablefield:col_1uniqueChecks if column values are uniquemodels:-name:my_tablecolumns:-name:col_1tests:-uniqueData Testsaccepted_valuesChecks if column values match a predefined list of accepted valuesmodels:-name:my_tablecolumns:-name:day_of_weekdescription:tests:-accepted_values:values:['Mon','Tue','Wed','Thu','Fri','Sat','Sun']at_least_oneChecks if column has at least one valuemodels:-name:my_tablecolumns:-name:col_1tests:-at_least_oneequal_expression (TBD)frequency (TBD)not_constantChecks if column has at more than one valuemodels:-name:my_tablecolumns:-name:col_1tests:-not_constantrecency (TBD)unique_rowsChecks if table rows are unique.
Ifcolumnsare not specified, uses all columns.models:-name:my_tabletests:-unique_rows:columns:[col_1,col_2,col_3]