Package: data-pipelines-cli
data-pipelines-cliCLI for data platformDocumentationRead the full documentation athttps://data-pipelines-cli.readthedocs.io/InstallationUse the package managerpipto installdp (data-pipelines-cli):pipinstalldata-pipelines-cli[bigquery,docker,datahub,gcs]UsageFirst, create a repository with a global configuration file that you or your organization will be using. The repository
should containdp.yml.tmplfile looking similar to this:_templates_suffix:".tmpl"_envops:autoescape:falseblock_end_string:"%]"block_start_string:"[%"comment_end_string:"#]"comment_start_string:"[#"keep_trailing_newline:truevariable_end_string:"]]"variable_start_string:"[["templates:my-first-template:template_name:my-first-templatetemplate_path:https://github.com/<YOUR_USERNAME>/<YOUR_TEMPLATE>.gitvars:username:[[YOUR_USERNAME]]Thanks to thecopier, you can leverage tmpl template syntax to create
easily modifiable configuration templates. Just create acopier.ymlfile next to thedp.yml.tmplone and configure
the template questions (read more atcopier documentation).Then, rundp init <CONFIG_REPOSITORY_URL>to initializedp. You can also drop<CONFIG_REPOSITORY_URL>argument,dpwill get initialized with an empty config.Project creationYou can usedp create <NEW_PROJECT_PATH>to choose one of the templates added before and create the project in the<NEW_PROJECT_PATH>directory. You can also usedp create <NEW_PROJECT_PATH> <LINK_TO_TEMPLATE_REPOSITORY>to point
directly to a template repository. If<LINK_TO_TEMPLATE_REPOSITORY>proves to be the name of the template defined indp's config file,dp createwill choose the template by the name instead of trying to download the repository.dp template-listlists all added templates.Project updateTo update your pipeline project usedp update <PIPELINE_PROJECT-PATH>. It will sync your existing project with updated
template version selected by--vcs-refoption (defaultHEAD).Project deploymentdp deploywill sync with your bucket provider. The provider will be chosen automatically based on the remote URL.
Usually, it is worth pointingdp deployto JSON or YAML file with provider-specific data like access tokens or project
names. E.g., to connect with Google Cloud Storage, one should run:echo'{"token": "<PATH_TO_YOUR_TOKEN>", "project_name": "<YOUR_PROJECT_NAME>"}'>gs_args.json
dpdeploy--dags-path"gs://<YOUR_GS_PATH>"--blob-argsgs_args.jsonHowever, in some cases you do not need to do so, e.g. when usinggcloudwith properly set local credentials. In such
case, you can try to run just thedp deploy --dags-path "gs://<YOUR_GS_PATH>"command. Please refer todocumentationfor more information.When finished, calldp cleanto remove compilation related directories.VariablesYou can put a dictionary of variables to be passed todbtin yourconfig/<ENV>/dbt.ymlfile, following the convention
presented inthe guide at the dbt site.
E.g., if one of the fields ofconfig/<SNOWFLAKE_ENV>/snowflake.ymllooks like this:schema:"{{var('snowflake_schema')}}"you should put the following in yourconfig/<SNOWFLAKE_ENV>/dbt.ymlfile:vars:snowflake_schema:EXAMPLE_SCHEMAand then run yourdp run --env <SNOWFLAKE_ENV>(or any similar command).ContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change.Please make sure to update tests as appropriate.