Package: datacontract-cli
Data Contract CLIThedatacontractCLI is an open source command-line tool for working withData Contracts.
It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.NOTE:This project has been migrated from Go to Python which adds the possibility to usedatacontractwithin Python code as library, but it comes with somebreaking changes. The Go version has beenforked, if you still rely on that.Getting startedLet's look at this data contract:https://datacontract.com/examples/orders-latest/datacontract.yamlWe have aserverssection with endpoint details to the S3 bucket,modelsfor the structure of the data, andqualityattributes that describe the expected freshness and number of rows.This data contract contains all information to connect to S3 and check that the actual data meets the defined schema and quality requirements. We can use this information to test if the actual data set in S3 is compliant to the data contract.Let's usepipto install the CLI (or use theDocker image, if you prefer).$python3-mpipinstalldatacontract-cliWe run the tests:$datacontracttesthttps://datacontract.com/examples/orders-latest/datacontract.yaml# returns: 🟢 data contract is valid. Run 22 checks.Voilà, the CLI tested that thedatacontract.yamlitself is valid, all records comply with the schema, and all quality attributes are met.Usage# create a new data contract from example and write it to datacontract.yaml$datacontractinitdatacontract.yaml# lint the datacontract.yaml$datacontractlintdatacontract.yaml# execute schema and quality checks$datacontracttestdatacontract.yaml# execute schema and quality checks on the examples within the contract$datacontracttest--examplesdatacontract.yaml# find differences between to data contracts (Coming Soon)$datacontractdiffdatacontract-v1.yamldatacontract-v2.yaml# fail pipeline on breaking changes  (Coming Soon)$datacontractbreakingdatacontract-v1.yamldatacontract-v2.yaml# export model as jsonschema$datacontractexport--formatjsonschemadatacontract.yaml# export model as dbt$datacontractexport--formatdbtdatacontract.yaml# import protobuf as model (Coming Soon)$datacontractimport--formatprotobuf--sourcemy_protobuf_file.protodatacontract.yamlProgrammatic (Python)fromdatacontract.data_contractimportDataContractdata_contract=DataContract(data_contract_file="datacontract.yaml")run=data_contract.test()ifnotrun.has_passed():print("Data quality validation failed.")# Abort pipeline, alert, or take corrective actions...Scenario: Integration with Data Mesh ManagerIf you useData Mesh Manager, you can use the data contract URL and append the--publishoption to send and display the test results. Set an environment variable for your API key.# Fetch current data contract, execute tests on production, and publish result to data mesh manager$EXPORTDATAMESH_MANAGER_API_KEY=xxx
$datacontracttesthttps://demo.datamesh-manager.com/demo279750347121/datacontracts/4df9d6ee-e55d-4088-9598-b635b2fdcbbc/datacontract.yaml--serverproduction--publishInstallationChoose the most appropriate installation method for your needs:pipPython 3.11 recommended.
Python 3.12 available as pre-release release candidate for 0.9.3python3-mpipinstalldatacontract-clipipxpipx installs into an isolated environment.pipxinstalldatacontract-cliDockerdockerpull--platformlinux/amd64datacontract/cli
dockerrun--rm--platformlinux/amd64-v${PWD}:/home/datacontractdatacontract/cliOr via an alias that automatically uses the latest version:aliasdatacontract='docker run --rm -v "${PWD}:/home/datacontract" --platform linux/amd64 datacontract/cli:latest'DocumentationTestsData Contract CLI can connect to data sources and run schema and quality tests to verify that the data contract is valid.$datacontracttest--serverproductiondatacontract.yamlTo connect to the databases theserverblock in the datacontract.yaml is used to set up the connection. In addition, credentials, such as username and passwords, may be defined with environment variables.The application uses different engines, based on the servertype.TypeFormatDescriptionStatusEnginess3parquetWorks for any S3-compliant endpoint., e.g., AWS S3, GCS, MinIO, Ceph, ...✅soda-core-duckdbs3jsonSupport fornew_linedelimited JSON files and one JSON record per file.✅fastjsonschemasoda-core-duckdbs3csv✅soda-core-duckdbs3deltaComing soonTBDpostgresn/a✅soda-core-postgressnowflaken/a✅soda-core-snowflakebigqueryn/a✅soda-core-bigqueryredshiftn/aComing soonTBDdatabricksn/aSupport for Databricks SQL with Unity catalog and Hive metastore.✅soda-core-sparkdatabricksn/aSupport for Spark for programmatic use in Notebooks.✅soda-core-spark-dfkafkajsonComing soonTBDkafkaavroComing soonTBDkafkaprotobufComing soonTBDlocalparquet✅soda-core-duckdblocaljsonSupport fornew_linedelimited JSON files and one JSON record per file.✅fastjsonschemasoda-core-duckdblocalcsv✅soda-core-duckdbFeel free to create an issue, if you need support for an additional type.S3Data Contract CLI can test data that is stored in S3 buckets or any S3-compliant endpoints in various formats.Exampledatacontract.yamlservers:production:type:s3endpointUrl:https://minio.example.com# not needed with AWS S3location:s3://bucket-name/path/*/*.jsonformat:jsondelimiter:new_line# new_line, array, or noneEnvironment VariablesEnvironment VariableExampleDescriptionDATACONTRACT_S3_REGIONeu-central-1Region of S3 bucketDATACONTRACT_S3_ACCESS_KEY_IDAKIAXV5Q5QABCDEFGHAWS Access Key IDDATACONTRACT_S3_SECRET_ACCESS_KEY93S7LRrJcqLaaaa/XXXXXXXXXXXXXAWS Secret Access KeyPostgresData Contract CLI can test data in Postgres or Postgres-compliant databases (e.g., RisingWave).Exampledatacontract.yamlservers:postgres:type:postgreshost:localhostport:5432database:postgresschema:publicmodels:my_table_1:# corresponds to a tabletype:tablefields:my_column_1:# corresponds to a columntype:varcharEnvironment VariablesEnvironment VariableExampleDescriptionDATACONTRACT_POSTGRES_USERNAMEpostgresUsernameDATACONTRACT_POSTGRES_PASSWORDmysecretpasswordPasswordBigQueryWe support authentication to BigQuery using Service Account Key. The used Service Account should include the roles:BigQuery Job UserBigQuery Data ViewerExampledatacontract.yamlservers:production:type:bigqueryproject:datameshexample-productdataset:datacontract_cli_test_datasetmodels:datacontract_cli_test_table:# corresponds to a BigQuery tabletype:tablefields:...Environment VariablesEnvironment VariableExampleDescriptionDATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH~/service-access-key.jsonService Access key as saved on key creation by BigQueryDatabricksWorks with Unity Catalog and Hive metastore.Needs a running SQL warehouse or compute cluster.Exampledatacontract.yamlservers:production:type:databrickshost:dbc-abcdefgh-1234.cloud.databricks.comcatalog:acme_catalog_prodschema:orders_latestmodels:orders:# corresponds to a tabletype:tablefields:...Environment VariablesEnvironment VariableExampleDescriptionDATACONTRACT_DATABRICKS_TOKENdapia00000000000000000000000000000The personal access token to authenticateDATACONTRACT_DATABRICKS_HTTP_PATH/sql/1.0/warehouses/b053a3ffffffffThe HTTP path to the SQL warehouse or compute clusterDatabricks (programmatic)Works with Unity Catalog and Hive metastore.
When running in a notebook or pipeline, the providedsparksession can be used.
An additional authentication is not required.Requires a Databricks Runtime with Python >= 3.10.Exampledatacontract.yamlservers:production:type:databrickshost:dbc-abcdefgh-1234.cloud.databricks.com# ignored, always use current hostcatalog:acme_catalog_prodschema:orders_latestmodels:orders:# corresponds to a tabletype:tablefields:...Notebook%pipinstalldatacontract-clidbutils.library.restartPython()fromdatacontract.data_contractimportDataContractdata_contract=DataContract(data_contract_file="/Volumes/acme_catalog_prod/orders_latest/datacontract/datacontract.yaml",spark=spark)run=data_contract.test()run.resultExportsAvailable export options:TypeDescriptionStatusjsonschemaExport to JSON Schema✅sodaclExport to SodaCL quality checks in YAML format✅dbtExport to dbt model in YAML format✅avroExport to AVRO modelsTBDpydanticExport to pydantic modelsTBDsqlExport to SQL DDLTBDprotobufExport to ProtobufTBDDevelopment SetupPython base interpreter should be 3.11.x (unless working on 3.12 release candidate).# create venvpython3-mvenvvenvsourcevenv/bin/activate# Install Requirementspipinstall--upgradepipsetuptoolswheel
pipinstall-e'.[dev]'cdtests/
pytestReleasegit tag v0.9.0
git push origin v0.9.0
python3 -m pip install --upgrade build twine
rm -r dist/
python3 -m build
# for now only test.pypi.org
python3 -m twine upload --repository testpypi dist/*Docker Builddocker build -t datacontract/cli .
docker run --rm -v ${PWD}:/home/datacontract datacontract/cliContributionWe are happy to receive your contributions. Propose your change in an issue or directly create a pull request with your improvements.LicenseMIT LicenseCreditsCreated byStefan NegeleandJochen Christ.<style>.github-corner:hover .octo-arm{animation:octocat-wave 560ms ease-in-out}@keyframes octocat-wave{0%,100%{transform:rotate(0)}20%,60%{transform:rotate(-25deg)}40%,80%{transform:rotate(10deg)}}@media (max-width:500px){.github-corner:hover .octo-arm{animation:none}.github-corner .octo-arm{animation:octocat-wave 560ms ease-in-out}}</style>