Repository: getindata/data-pipelines-cli (language: python)

Issue:
Release 0.21.0
Bump version and CHANGELOG for next release.


Discussion / hints:

Fix patch:
diff --git a/.gitignore b/.gitignore
index 8192175..d3b6707 100644
--- a/.gitignore
+++ b/.gitignore
@@ -124,3 +124,5 @@ venv.bak/
 .mypy_cache/
 
 docs/_build
+
+dp-testing
\ No newline at end of file
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 30ebea4..f76fb63 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,10 @@
 
 ## [Unreleased]
 
+## [0.21.0] - 2022-07-19
+
+-   Documentation improvements
+
 ## [0.20.1] - 2022-06-17
 
 ### Fixed
@@ -207,7 +211,9 @@
 -   Draft of `dp init`, `dp create`, `dp template new`, `dp template list` and `dp dbt`
 -   Draft of `dp compile` and `dp deploy`
 
-[Unreleased]: https://github.com/getindata/data-pipelines-cli/compare/0.20.1...HEAD
+[Unreleased]: https://github.com/getindata/data-pipelines-cli/compare/0.21.0...HEAD
+
+[0.21.0]: https://github.com/getindata/data-pipelines-cli/compare/0.20.1...0.21.0
 
 [0.20.1]: https://github.com/getindata/data-pipelines-cli/compare/0.20.0...0.20.1
 
diff --git a/data_pipelines_cli/__init__.py b/data_pipelines_cli/__init__.py
index a8e7d39..8edcc83 100644
--- a/data_pipelines_cli/__init__.py
+++ b/data_pipelines_cli/__init__.py
@@ -5,4 +5,4 @@
 pipelines.
 """
 
-version = "0.20.1"
+version = "0.21.0"
diff --git a/docs/configuration.rst b/docs/configuration.rst
new file mode 100644
index 0000000..1cfa1e7
--- /dev/null
+++ b/docs/configuration.rst
@@ -0,0 +1,168 @@
+Project configuration
+===========
+
+**dp** as a tool depends on a few files in your project directory. It must be able to find a ``config`` directory with
+a structure looking similar to this:
+
+| config
+| ├── base
+| │   ├── dbt.yml
+| │   ├── bigquery.yml
+| │   └── ...
+| ├── dev
+| │   └── bigquery.yml
+| │── local
+| │   ├── dbt.yml
+| │   └── bigquery.yml
+| └── prod
+|     └── bigquery.yml
+|
+
+Whenever you call **dp**'s command with the ``--env <ENV>`` flag, the tool will search for ``dbt.yml`` and
+``<TARGET_TYPE>.yml`` files in ``base`` and ``<ENV>`` directory and parse important info out of them, with ``<ENV>``
+settings taking precedence over those listed in ``base``. So, for example, for the following files:
+
+.. code-block:: yaml
+
+ # config/base/dbt.yml
+ target: env_execution
+ target_type: bigquery
+
+ # config/base/bigquery.yml
+ method: oauth
+ project: my-gcp-project
+ dataset: my-dataset
+ threads: 1
+
+ # cat config/dev/bigquery.yml
+ dataset: dev-dataset
+
+``dp test --env dev`` will run ``dp test`` command using values from those files, most notably with ``dataset: dev-dataset`` overwriting
+``dataset: my-dataset`` setting.
+
+**dp** synthesizes dbt's ``profiles.yml`` out of those settings among other things. However, right now it only creates
+``local`` or ``env_execution`` profile, so if you want to use different settings amongst different environments, you
+should rather use ``{{ env_var('VARIABLE') }}`` as a value and provide those settings as environment variables. E.g., by
+setting those in your ``config/<ENV>/k8s.yml`` file, in ``envs`` dictionary:
+
+.. code-block:: yaml
+
+ # config/base/bigquery.yml
+ method: oauth
+ dataset: "{{ env_var('GCP_DATASET') }}"
+ project: my-gcp-project
+ threads: 1
+
+ # config/base/execution_env.yml
+ # ... General config for execution env ...
+
+ # config/base/k8s.yml
+ # ... Kubernetes settings ...
+
+ # config/dev/k8s.yml
+ envs:
+   GCP_DATASET: dev-dataset
+
+ # config/prod/k8s.yml
+ envs:
+    GCP_DATASET: prod-dataset
+
+dbt configuration
+++++++++++++++++++++++++++++++
+
+The main configuration is in ``config/<ENV>/dbt.yml`` file. At the moment it allows setting two values:
+* ``target`` - should be set either to ``local`` or ``env_execution`` depending on where the tool is used. Local means
+running locally while ``env_execution`` means executing by the scheduler on the dev or prod environment.
+* ``target_type`` - defines which backend **dbt** will use and what file **dp** will search for additional configuration
+(example: ``bigquery`` or ``snowflake``).
+
+Additionally, the backend configuration file should be provided with a name depending on the selected ``target_type``
+(<target_type>.yml). For example setting ``target_type`` to ``bigquery`` **dp** will look for bigquery.yml files.
+This file should consist of all configurations that will be used to build profile.yml. Example files for the production
+environment:
+
+.. code-block:: yaml
+
+   method: service-account
+   keyfile: "{{ env_var('GCP_KEY_PATH') }}"
+   project: gid-dataops-labs
+   dataset: presentation
+   threads: 1
+   timeout_seconds: 300
+   priority: interactive
+   location: europe-central2
+   retries: 1
+
+Variables
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can put a dictionary of variables to be passed to ``dbt`` in your ``config/<ENV>/dbt.yml`` file, following the convention
+presented in `the guide at the dbt site <https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#defining-variables-in-dbt_projectyml>`_.
+E.g., if one of the fields of ``config/<SNOWFLAKE_ENV>/snowflake.yml`` looks like this:
+
+.. code-block:: yaml
+
+ schema: "{{ var('snowflake_schema') }}"
+
+you should put the following in your ``config/<SNOWFLAKE_ENV>/dbt.yml`` file:
+
+.. code-block:: yaml
+
+ vars:
+   snowflake_schema: EXAMPLE_SCHEMA
+
+and then run your ``dp run --env <SNOWFLAKE_ENV>`` (or any similar command).
+
+You can also add "global" variables to your **dp** config file ``$HOME/.dp.yml``. Be aware, however, that those variables
+get erased on every ``dp init`` call. It is a great idea to put *commonly used* variables in your organization's
+``dp.yml.tmpl`` template and make **copier** ask for those when initializing **dp**. By doing so, each member of your
+organization will end up with a list of user-specific variables reusable across different projects on its machine.
+Just remember, **global-scoped variables take precedence over project-scoped ones.**
+
+Airflow configuration
+++++++++++++++++++++++++++++++
+
+Airflow-related configuration is stored in ``config/<ENV>/airflow.yml`` file and is strongly connected to the Airflow plugin: ``dbt-airflow-factory``
+More information about this configuration can be found `here <https://dbt-airflow-factory.readthedocs.io/en/latest/configuration.html#airflow-yml-file>`_
+
+One important config from **dp** tool in this file is ``dags_path``. It sets the URL to blob storage that is responsible for
+storing projects DAGs with other artifacts.
+
+Execution environment configuration
+++++++++++++++++++++++++++++++
+
+All configuration about how **dbt** is executed on the Airflow side is kept in execution_env.yml and <env type>.yml. More
+information about these settings can be found `here <https://dbt-airflow-factory.readthedocs.io/en/latest/configuration.html#execution-env-yml-file>`_
+
+Publication configuration
+++++++++++++++++++++++++++++++
+
+``config/<ENV>/publish.yml`` file contains configuration about creating **dbt** packages for downstream projects and
+publishing it to a git repository as a package registry.
+
+.. list-table::
+   :widths: 25 20 55
+   :header-rows: 1
+
+   * - Parameter
+     - Data type
+     - Description
+   * - repository
+     - string
+     - HTTPS link to repo that works as packages repository.
+   * - branch
+     - string
+     - Branch of the selected repository where packages are published.
+   * - username
+     - string
+     - User name that will be presented as package publisher in GIT.
+   * - email
+     - string
+     - Email of the package publisher.
+
+Data governance configuration
+++++++++++++++++++++++++++++++
+
+**dp** can sends **dbt** metadata to DataHub. All related configuration is stored in ``config/<ENV>/datahub.yml`` file.
+More information about it can be found `here <https://datahubproject.io/docs/metadata-ingestion#recipes>`_ and `here <https://datahubproject.io/docs/generated/ingestion/sources/dbt>`_.
+
diff --git a/docs/images/created.png b/docs/images/created.png
new file mode 100644
index 0000000..5fb4002
Binary files /dev/null and b/docs/images/created.png differ
diff --git a/docs/images/creating.png b/docs/images/creating.png
new file mode 100644
index 0000000..d1f3528
Binary files /dev/null and b/docs/images/creating.png differ
diff --git a/docs/images/init.png b/docs/images/init.png
new file mode 100644
index 0000000..b7018fd
Binary files /dev/null and b/docs/images/init.png differ
diff --git a/docs/images/integration.png b/docs/images/integration.png
new file mode 100644
index 0000000..8e1c101
Binary files /dev/null and b/docs/images/integration.png differ
diff --git a/docs/images/railsroad.png b/docs/images/railsroad.png
new file mode 100644
index 0000000..285e8e3
Binary files /dev/null and b/docs/images/railsroad.png differ
diff --git a/docs/images/run.png b/docs/images/run.png
new file mode 100644
index 0000000..d4f7a67
Binary files /dev/null and b/docs/images/run.png differ
diff --git a/docs/index.rst b/docs/index.rst
index 6a81934..920c27f 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -1,4 +1,4 @@
-``data-pipelines-cli``: CLI for data platform
+``Data Pipelines CLI``: CLI for data platform
 ==============================================
 
 .. image:: https://img.shields.io/badge/python-3.7%20%7C%203.8%20%7C%203.9-blue.svg
@@ -21,22 +21,32 @@
    :target: https://codeclimate.com/github/getindata/data-pipelines-cli/test_coverage
    :alt: Test Coverage
 
-
-Installation
+Introduction
 ------------
-Use the package manager `pip <https://pip.pypa.io/en/stable/>`_ to
-install `dp (data-pipelines-cli) <https://pypi.org/project/data-pipelines-cli/>`_:
+**Data Pipelines CLI**, also called **DP tool**, is a command-line tool providing an easy way to build and manage data pipelines based on **dbt**
+in an environment with GIT, Airflow, DataHub, VSCode, etc.
 
-.. code-block:: sh
+The tool can be used in any environment with access to shell and ``Python`` installed.
 
-   pip install data-pipelines-cli
+**data-pipelines-cli**'s main task is to cover technical complexities and provides an abstraction over all components
+that take part in Data Pipelines creation and execution. Thanks to the integration with templating engine it allows Analytics
+Engineers to create and configure new projects. The tool also simplifies automation as it handles deployments and
+publications of created transformations.
 
+Community
+------------
+Although the tools were created by `GetInData <https://getindata.com/>`_ and used in their project it is open-sourced and everyone is welcome
+to use and contribute to making it better and more powerful.
 
 .. toctree::
    :maxdepth: 1
    :caption: Contents:
 
+   installation
+   setup_environment
    usage
+   configuration
+   integration
    cli
    api
    changelog
diff --git a/docs/installation.md b/docs/installation.md
new file mode 100644
index 0000000..eb4c029
--- /dev/null
+++ b/docs/installation.md
@@ -0,0 +1,12 @@
+# Installation
+------------
+Use the package manager `pip <https://pip.pypa.io/en/stable/>`_ to
+install `dp (data-pipelines-cli) <https://pypi.org/project/data-pipelines-cli/>`_:
+
+```shell
+pip install data-pipelines-cli
+```
+
+## Compatibility
+
+The library is compatibly with Python >= 3.7.
diff --git a/docs/integration.rst b/docs/integration.rst
new file mode 100644
index 0000000..66ed7e9
--- /dev/null
+++ b/docs/integration.rst
@@ -0,0 +1,72 @@
+Integration with environment
+======================
+
+**Data Pipelines CLI** provides some sort of abstraction over multiple other components that take part in Data Pipeline
+processes. The following picture presents the whole environment which is handled by our tool.
+
+.. image:: images/integration.png
+   :width: 700
+
+dbt
+++++++++++++++++++++++++++++++++++++++++++++++
+
+`dbt <https://www.getdbt.com/>`_ is currently the main tool that **DP** integrates with. The purpose of the **DP** tool is to cover **dbt** technicalities
+including configuration and generates it on the fly whenever needed. At the same time, it gives more control over **dbt**
+process management by chaining commands, interpolating configuration, and providing easy environments portability.
+
+Copier
+++++++++++++++++++++++++++++++++++++++++++++++
+
+**DP** is heavily using `Copier <https://copier.readthedocs.io/en/stable/>`_ as templating tool. It gives a possibility to easily create new projects that are configured
+automatically after a series of questions. It is also used to configure the working environment with required environment
+variables.
+
+Docker
+++++++++++++++++++++++++++++++++++++++++++++++
+
+One of the artifacts during building and publishing Data Pipelines are `Docker's <https://www.docker.com/>`_ images. Each
+created image contains **dbt** with its transformation and scripts to run. Created images are environment agnostic and
+can be deployed in any external configuration. Images are pushed to the selected Container Registry which configuration
+should be taken from the environment (there should be a docker client configured).
+
+Git
+++++++++++++++++++++++++++++++++++++++++++++++
+
+The `Data Pipelines CLI` can also publish created **dbt** packages for downstream usage into configured
+`GIT <https://git-scm.com/>`_ repository. It uses key-based authentication where the key is provided as parameter `--key-path`
+
+Airflow
+++++++++++++++++++++++++++++++++++++++++++++++
+
+**DP** doesn't communicate directly with Airflow, it rather sends artifacts to Object storage managed by Airflow and
+`dbt-airflow-factory <https://dbt-airflow-factory.readthedocs.io/en/latest/>`_ library handles the rest. Created
+projects keep DAG and configuration required to execute on the Airflow side.
+
+Object storage
+++++++++++++++++++++++++++++++++++++++++++++++
+
+Configuration, Airflow DAG, and **dbt** manifest.json file are stored in Object storage for Airflow to be picked up and executed.
+the **DP** uses `fsspec <https://filesystem-spec.readthedocs.io/en/latest/>`_ which gives a good abstraction over different
+object storage providers. Currently, the tools were tested with GCS and S3.
+
+DataHub
+++++++++++++++++++++++++++++++++++++++++++++++
+
+The `Data Pipelines CLI` is able to send data to `DataHub <https://datahubproject.io/>`_ based on a recipe in configuration.
+The tool uses DataHub CLI under the hoot.
+
+Visual Studio Code
+++++++++++++++++++++++++++++++++++++++++++++++
+
+`VS Code <https://code.visualstudio.com/>`_ is one of the recommended by us tools to work with **dbt**. **DP** tool simplify
+integration of the created project with the `VS Code` plugin for **dbt** management.
+
+Airbyte
+++++++++++++++++++++++++++++++++++++++++++++++
+
+Under development
+
+Looker
+++++++++++++++++++++++++++++++++++++++++++++++
+
+Under development
diff --git a/docs/requirements.txt b/docs/requirements.txt
index 6ecdde1..fbaf2d6 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -3,3 +3,6 @@ sphinx-click>=4.0,<4.1
 myst-parser>=0.17, <0.18
 docutils>=0.17,<0.18
 GitPython==3.1.26
+colorama==0.4.4
+MarkupSafe<2.1
+Werkzeug==2.0.3
\ No newline at end of file
diff --git a/docs/setup_environment.rst b/docs/setup_environment.rst
new file mode 100644
index 0000000..b6576d4
--- /dev/null
+++ b/docs/setup_environment.rst
@@ -0,0 +1,41 @@
+Setup an environment
+=====
+
+This section is for Data Engineers who will be preparing and administrating the whole environment.
+It describes steps that should be done to prepare the DP tool to be used in an organization with full potential.
+
+Create Data Pipeline project template
+----------------
+
+The first thing that you need to do is to create a git repository with a project template used later to create multiple
+projects. The template should contain the whole directory structure and files used in your projects.
+Additionally, it should have a connection configuration to all components in your environment, CICD, and all other
+aspects specific to your company. Here you can find templates examples that you can adjust to your need:
+https://github.com/getindata/data-pipelines-template-example . Based on the template The Data Pipelines CLI will ask a user
+a series of questions to build the final project.
+
+Thanks to the ``copier`` you can leverage Jinja template syntax to create easily modifiable configuration templates.
+Just create a ``copier.yml`` and configure the template questions (read more at
+`copier documentation <https://copier.readthedocs.io/en/stable/configuring/>`_).
+
+Create a template to setup a local environment
+----------------
+
+Working with Data Pipelines usually requires local variables to be set to run and test avoiding messing in shared environments (DEV, STAGE, PROD). To simplify working environment preparation we also
+decided to use templates that will ask a series of questions and generate local configuration in a home directory.
+
+It requires a repository with a global configuration template file that you or your organization will be using.
+The repository should contain ``dp.yml.tmpl`` file looking similar to this:
+
+.. code-block:: yaml
+
+ templates:
+   my-first-template:
+     template_name: my-first-template
+     template_path: https://github.com/<YOUR_USERNAME>/<YOUR_TEMPLATE>.git
+ vars:
+   username: YOUR_USERNAME
+
+The file must contain a list of available templates. The templates will be displayed and available for selection in
+Data Pipelines CLI. The next section contains variables that will be passed to the project whenever running in the configured environment. The
+same rules apply in template creation as for project templates.
\ No newline at end of file
diff --git a/docs/usage.rst b/docs/usage.rst
index 003d686..9a7383a 100644
--- a/docs/usage.rst
+++ b/docs/usage.rst
@@ -1,145 +1,133 @@
 Usage
 =====
 
-First, create a repository with a global configuration file that you or your organization will be using. The repository
-should contain ``dp.yml.tmpl`` file looking similar to this:
+This section is for ``Data Pipelines CLI``'s users. It will present how to use the tool and how it handles interaction
+with the whole Data environment. Below diagram presents the sequence process how usually the toole is used and order in
+which different commands are executed:
 
-.. code-block:: yaml
+.. image:: images/railsroad.png
+   :width: 700
+
+Preparing working environment
+-------------------------
 
- templates:
-   my-first-template:
-     template_name: my-first-template
-     template_path: https://github.com/<YOUR_USERNAME>/<YOUR_TEMPLATE>.git
- vars:
-   username: YOUR_USERNAME
+The first thing that needs to be done when starting Building Data Pipelines is to prepare the working environment. This step
+can be done either on a local machine on any kind of Workbench (eg. JupyterLab). You will need a link from your
+Data Engineer or Administrator to the template with initial configuration then, run ``dp init <CONFIG_REPOSITORY_URL>``
+to initialize **dp**. You can also drop ``<CONFIG_REPOSITORY_URL>`` argument, **dp** will get initialized with an empty
+config.
 
-Thanks to the `copier <https://copier.readthedocs.io/en/stable/>`_, you can leverage Jinja template syntax to create
-easily modifiable configuration templates. Just create a ``copier.yml`` file next to the ``dp.yml.tmpl`` one and configure
-the template questions (read more at `copier documentation <https://copier.readthedocs.io/en/stable/configuring/>`_).
+This step is done only the first time for each working environment you want to use.
 
-Then, run ``dp init <CONFIG_REPOSITORY_URL>`` to initialize **dp**. You can also drop ``<CONFIG_REPOSITORY_URL>`` argument,
-**dp** will get initialized with an empty config.
+Example:
+
+In this example only one variable you will be asked for and it is going to be `username` which is used in many dp commands.
+
+.. code-block:: bash
+
+ dp init https://github.com/getindata/data-pipelines-cli-init-example
+
+
+.. image:: images/init.png
+   :width: 700
 
 Project creation
 ----------------
 
-You can use ``dp create <NEW_PROJECT_PATH>`` to choose one of the templates added before and create the project in the
+You can use ``dp create <NEW_PROJECT_PATH>`` to choose one of the templates to create the project in the
 ``<NEW_PROJECT_PATH>`` directory.
 
 You can also use ``dp create <NEW_PROJECT_PATH> <LINK_TO_TEMPLATE_REPOSITORY>`` to point directly to a template
 repository. If ``<LINK_TO_TEMPLATE_REPOSITORY>`` proves to be the name of the template defined in **dp**'s config file,
 ``dp create`` will choose the template by the name instead of trying to download the repository.
 
-``dp template-list`` lists all added templates.
+After the template selection, you will be asked a series of predefined questions in the template.  Answering them all will cause
+a new empty project to be generated. The project will be adjusted and personalized based on answers to the questions.
 
-Project update
---------------
+Example:
 
-To update your pipeline project use ``dp update <PIPELINE_PROJECT-PATH>``. It will sync your existing project with updated
-template version selected by ``--vcs-ref`` option (default ``HEAD``).
-
-Project configuration
----------------------
-
-**dp** as a tool depends on a few files in your project directory. In your project directory, it must be able to find a
-``config`` directory with a structure looking similar to this:
-
-| config
-| ├── base
-| │   ├── dbt.yml
-| │   ├── bigquery.yml
-| │   └── ...
-| ├── dev
-| │   └── bigquery.yml
-| │── local
-| │   ├── dbt.yml
-| │   └── bigquery.yml
-| └── prod
-|     └── bigquery.yml
-|
+Following command starts project creation process.
 
-Whenever you call **dp**'s command with the ``--env <ENV>`` flag, the tool will search for ``dbt.yml`` and
-``<TARGET_TYPE>.yml`` files in ``base`` and ``<ENV>`` directory and parse important info out of them, with ``<ENV>``
-settings taking precedence over those listed in ``base``. So, for example, for the following files:
+.. code-block:: bash
 
-.. code-block:: yaml
+ dp create our-simple-project
 
- # config/base/dbt.yml
- target: env_execution
- target_type: bigquery
+Fist step after this command is template selection:
 
- # config/base/bigquery.yml
- method: oauth
- project: my-gcp-project
- dataset: my-dataset
- threads: 1
+.. image:: images/creating.png
+   :width: 700
 
- # cat config/dev/bigquery.yml
- dataset: dev-dataset
+We can switch options by pressing up and down buttons and we can make a decision by pressing enter.
+After that, series of questions will be asked. Be aware that the name of the DP project should be composed of alpha-numeric
+signs and the _ sign. After answering these questions the tool will generate complete project.
 
-``dp test --env dev`` will run ``dp test`` command using values from those files, most notably with ``dataset: dev-dataset`` overwriting
-``dataset: my-dataset`` setting.
+.. image:: images/created.png
+   :width: 700
 
-**dp** synthesizes dbt's ``profiles.yml`` out of those settings among other things. However, right now it only creates
-``local`` or ``env_execution`` profile, so if you want to use different settings amongst different environments, you
-should rather use ``{{ env_var('VARIABLE') }}`` as a value and provide those settings as environment variables. E.g., by
-setting those in your ``config/<ENV>/k8s.yml`` file, in ``envs`` dictionary:
 
-.. code-block:: yaml
+Adapting working environment to VSCode
+-------------------------
 
- # config/base/bigquery.yml
- method: oauth
- dataset: "{{ env_var('GCP_DATASET') }}"
- project: my-gcp-project
- threads: 1
+VSCode is recommended tool to work with **dbt** as you can add a plugin that makes the work more efficient. To configure
+the plugin or integrate it with some other standalone application you will need to generate ``profiles.yml`` file from the project.
+``dp prepare-env`` prepares your local environment to be more conformant with standalone **dbt** requirements by saving
+``profiles.yml`` in the home directory.
 
- # config/base/execution_env.yml
- # ... General config for execution env ...
+However, be aware that IDE usage is optional, and you can comfortably use ``dp run`` and ``dp test``
+commands to interface with the **dbt** instead.
 
- # config/base/k8s.yml
- # ... Kubernetes settings ...
+List all available templates
+----------------
 
- # config/dev/k8s.yml
- envs:
-   GCP_DATASET: dev-dataset
+Execute ``dp template-list`` to list all added templates.
 
- # config/prod/k8s.yml
- envs:
-    GCP_DATASET: prod-dataset
+Project update
+--------------
 
-``target`` and ``target_type``
-++++++++++++++++++++++++++++++
+Whenever the template change you can update your project using ``dp update <PIPELINE_PROJECT-PATH>`` command.
+It will sync your existing project with the updated template version selected by ``--vcs-ref`` option (default ``HEAD``).
 
-* ``target`` setting in ``config/<ENV>/dbt.yml`` should be set either to ``local`` or ``env_execution``;
-* ``target_type`` defines which backend dbt will use and what file **dp** will search for; example ``target_types`` are ``bigquery`` or ``snowflake``.
+It may be very useful when the are some infrastructure changes in your organization and you need to upgrade all created
+projects (there can be hundreds of them).
 
-Variables
-+++++++++
+Project compilation
+-------------------
 
-You can put a dictionary of variables to be passed to ``dbt`` in your ``config/<ENV>/dbt.yml`` file, following the convention
-presented in `the guide at the dbt site <https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables#defining-variables-in-dbt_projectyml>`_.
-E.g., if one of the fields of ``config/<SNOWFLAKE_ENV>/snowflake.yml`` looks like this:
+``dp compile`` prepares your project to be run on your local machine and/or deployed on a remote one.
 
-.. code-block:: yaml
+Local run
+---------
 
- schema: "{{ var('snowflake_schema') }}"
+When you get your project created, you can run ``dp run`` and ``dp test`` commands.
 
-you should put the following in your ``config/<SNOWFLAKE_ENV>/dbt.yml`` file:
+* ``dp run`` runs the project on your local machine,
+* ``dp test`` run tests for your project on your local machine.
 
-.. code-block:: yaml
+Both commands accept ``--env`` parameter to select the execution environment. The default value is ``local``.
+
+Example:
+
+.. code-block:: bash
+
+ dp run
+
+This process will look at the contents of the models directory and create coresponding tables or views in data storage.
+
+.. image:: images/run.png
+   :width: 700
+
+Now after all the tables and views are created we can also check, if the models work as intended by running the tests.
+
+.. code-block:: bash
+
+ dp test
 
- vars:
-   snowflake_schema: EXAMPLE_SCHEMA
 
-and then run your ``dp run --env <SNOWFLAKE_ENV>`` (or any similar command).
+.. image:: images/test.png
+   :width: 700
 
-You can also add "global" variables to your **dp** config file ``$HOME/.dp.yml``. Be aware, however, that those variables
-get erased on every ``dp init`` call. It is a great idea to put *commonly used* variables in your organization's
-``dp.yml.tmpl`` template and make **copier** ask for those when initializing **dp**. By doing so, each member of your
-organization will end up with a list of user-specific variables reusable across different projects on its machine.
-Just remember, **global-scoped variables take precedence over project-scoped ones.**
 
-dbt sources and models creation
+dbt sources and automatic models creation
 -------------------------------
 
 With the help of `dbt-codegen <https://hub.getdbt.com/dbt-labs/codegen/>`_ and
@@ -186,23 +174,16 @@ If you add the **dbt-profiler** package to your ``packages.yml`` file too, you c
 ``dp generate model-yaml --with-meta MODELS_DIR``. **dbt-profiler** will add a lot of profiling metadata to
 descriptions of your models.
 
-Project compilation
--------------------
-
-``dp compile`` prepares your project to be run on your local machine and/or deployed on a remote one.
-
-Local run
----------
-
-When you get your project configured, you can run ``dp run`` and ``dp test`` commands.
-
-* ``dp run`` runs the project on your local machine,
-* ``dp test`` run tests for your project on your local machine.
-
 Project deployment
 ------------------
 
-``dp deploy`` will sync with your bucket provider. The provider will be chosen automatically based on the remote URL.
+``dp deploy`` executes the deployment of a project. Depending on the configuration the command may execute different steps
+described in this section. Please be aware that this command is meant for the CICD process and usually should be avoided as manual activity.
+
+Blob storage synchronization
+++++++++++++++++++++++++++++++++
+
+The main action of the ``dp deploy`` command is synchronization with your bucket provider. The provider will be chosen automatically based on the remote URL.
 Usually, it is worth pointing ``dp deploy`` to a JSON or YAML file with provider-specific data like access tokens or project
 names. The *provider-specific data* should be interpreted as the ``**kwargs`` (keyword arguments) expected by a specific
 `fsspec <https://filesystem-spec.readthedocs.io/en/latest/>`_'s FileSystem implementation. One would most likely want to
@@ -222,10 +203,7 @@ the credentials.
 Please refer to the documentation of the specific ``fsspec``'s implementation for more information about the required
 keyword arguments.
 
-``dags-path`` as config argument
-++++++++++++++++++++++++++++++++
-
-You can also list your path in the ``config/base/airflow.yml`` file, as a ``dags_path`` argument:
+You can also provide your path in the ``config/base/airflow.yml`` file, as a ``dags_path`` argument:
 
 .. code-block:: yaml
 
@@ -234,23 +212,67 @@ You can also list your path in the ``config/base/airflow.yml`` file, as a ``dags
 
 In such a case, you do not have to provide a ``--dags-path`` flag, and you can just call ``dp deploy`` instead.
 
+Docker image
+++++++++++++++++++++++++++++++++
+
+``dp deploy`` command builds Docker image with **dbt** and project and sends it go Docker Registry. Docker registry may be
+configured via Environment Variables (eg. DOCKER_AUTH_CONFIG) and the image repository can be configured in
+``execution_env.yml`` file. Use ``--docker-push`` flag to enable docker pushing during deployment.
+
+DataHub synchronization
+++++++++++++++++++++++++++++++++
+
+The deployment also sends metadata to ``DataHub`` based on receipt created in ``datahub.yml`` file. Use ``--datahub-ingest``
+flag to enable DataHub synchronization.
+
 Packing and publishing
 ----------------------
 
-The built project can be processed to a **dbt** package by calling ``dp publish``. ``dp publish`` parses ``manifest.json``
-and prepares a package that lists models outputted by transformations, saving it in the ``build/package`` directory.
+Sometimes there is a need to reuse data created in other projects and/or by a different team. The built project can be
+converted to a **dbt** package by calling ``dp publish``. ``dp publish`` parses ``manifest.json``
+and prepares a package from the presentation layer. It lists models created by transformations and they usually are a final product of a project. The models are prepared in form of **dbt** sources. Created metadata files are saved in the ``build/package`` directory and sent to a git repository
+configured in ``publish.yml`` file.
 
-Preparing dbt environment
--------------------------
+Publication repo usually is private for a company and appropriate permissions are required. We recommend key-based
+communication. You can use ``--key-path`` as a parameter to point to the key file with push permissions.
 
-Sometimes you would like to use standalone **dbt** or an application that interfaces with it (like VS Code plugin).
-``dp prepare-env`` prepares your local environment to be more conformant with standalone **dbt** requirements, e.g.,
-by saving ``profiles.yml`` in the home directory.
+Using published sources
+++++++++++++++++++++++++++++++++
 
-However, be aware that most of the time you do not need to do so, and you can comfortably use ``dp run`` and ``dp test``
-commands to interface with the **dbt** instead.
+Published packages can be used as standard **dbt** packages by adding them in ``packages.yml`` in the following form:
+
+.. code-block:: yaml
+
+ packages:
+  - git: "https://{{env_var('DBT_GIT_USER_NAME', '')}}:{{env_var('DBT_GIT_SECRET_TOKEN', '')}}@gitlab.com/<path to you repository>"
+    subdirectory: "<upstream project name>"
+
+Dependencies metadata
+++++++++++++++++++++++++++++++++
+
+Created metadata files containing extra information about the project name (which can be also Airflow DAG name).
+
+.. code-block:: json
+
+ "source_meta": {
+    "dag": "<project name>"
+ }
+
+This way explicit dependencies can be created in the execution environment. For more information see the documentation of
+`dbt-airflow-factory <https://dbt-airflow-factory.readthedocs.io/en/latest/features.html#source-dependencies>`
 
 Clean project
 -------------
 
-When finished, call ``dp clean`` to remove compilation-related directories.
+If needed call ``dp clean`` to remove compilation-related directories.
+
+Load seed
+-------------
+
+One can use ``dp seed`` to load seeds from the project. Use ``--env`` to choose a different environment.
+
+Serve documentation
+-------------
+
+dbt creates quite good documentation and sometimes it is useful to expose them to your coworkers on a custom port. To do that you can run
+``dbt docs --port <port>`` command.
diff --git a/setup.cfg b/setup.cfg
index 61a3a0f..f25582f 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -1,5 +1,5 @@
 [bumpversion]
-current_version = 0.20.1
+current_version = 0.21.0
 
 [bumpversion:file:setup.py]
 
diff --git a/setup.py b/setup.py
index 5ea3e3a..082fa2f 100644
--- a/setup.py
+++ b/setup.py
@@ -6,8 +6,9 @@
     README = f.read()
 
 INSTALL_REQUIREMENTS = [
-    "MarkupSafe==2.0.1",  # https://markupsafe.palletsprojects.com/en/2.1.x/changes/#version-2-1-0
-    "dbt-core==1.0.4",
+    "MarkupSafe<2.1",
+    "Werkzeug==2.0.3",
+    "dbt-core==1.1.1",
     "click>=8.0.3,<9.0",
     "questionary==1.10.0",
     "pyyaml>=5.1, <6.0",
@@ -16,6 +17,7 @@
     "Jinja2>=2.11,<2.12",
     "fsspec",
     "packaging>=20.4,<21.0",
+    "colorama==0.4.4",
 ]
 
 EXTRA_FILESYSTEMS_REQUIRE = {
@@ -25,13 +27,13 @@
 
 EXTRA_REQUIRE = {
     # DBT adapters
-    "bigquery": ["dbt-bigquery==1.0.0"],
-    "postgres": ["dbt-postgres==1.0.4"],
-    "snowflake": ["dbt-snowflake==1.0.0"],
-    "redshift": ["dbt-redshift==1.0.0"],
+    "bigquery": ["dbt-bigquery==1.1.1"],
+    "postgres": ["dbt-postgres==1.1.1"],
+    "snowflake": ["dbt-snowflake==1.1.0"],
+    "redshift": ["dbt-redshift==1.1.0"],
     # ---
     "docker": ["docker>=5.0"],
-    "datahub": ["acryl-datahub>=0.8.17, <0.8.18"],
+    "datahub": ["acryl-datahub[dbt]>=0.8.39"],
     "git": ["GitPython==3.1.26"],
     "tests": [
         "pytest>=6.2.2, <7.0.0",
@@ -55,13 +57,14 @@
         "sphinx-click>=4.0,<4.1",
         "myst-parser>=0.17, <0.18",
         "GitPython==3.1.26",
+        "colorama==0.4.4",
     ],
     **EXTRA_FILESYSTEMS_REQUIRE,
 }
 
 setup(
     name="data_pipelines_cli",
-    version="0.20.1",
+    version="0.21.0",
     description="CLI for data platform",
     long_description=README,
     long_description_content_type="text/markdown",


Test patch:
diff --git a/docs/images/test.png b/docs/images/test.png
new file mode 100644
index 0000000..5d0f227
Binary files /dev/null and b/docs/images/test.png differ