Repository: astronomer/astro-provider-databricks (language: python)

Issue:
Improve the README
▪︎ Installation
▪︎ Example or link to example DAGs
▪︎ Link to the changelog
▪︎ License
Enable code coverage
ATM we are missing Code Coverage, we should set it up and enable it in our CI.


Discussion / hints:

Fix patch:
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index b0b2048..94b9bca 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -112,7 +112,6 @@ jobs:
             ~/.cache/pip
             .nox
           key: unit-tests-os-${{ runner.os }}-python-${{ matrix.python }}-airflow-${{ matrix.airflow }}-deps-${{ hashFiles('pyproject.toml') }}-version-${{ hashFiles('src/astro_databricks/__init__.py') }}
-      - run: sqlite3 /tmp/sqlite_default.db "VACUUM;"
       - run: pip3 install nox
       - run: nox -s "test-${{ matrix.python }}(airflow='${{ matrix.airflow }}')" -- --ignore "tests/test_example_dags.py" --cov-report=xml
       - name: Upload coverage
@@ -127,6 +126,7 @@ jobs:
       matrix:
         python: [ '3.8', '3.9', '3.10' ]
         airflow: [ '2.2.4', '2.3', '2.4', '2.5' ]
+
     if: >-
       github.event_name == 'push' ||
       (
diff --git a/CHANGELOG.rst b/CHANGELOG.rst
index f7b80c6..5bbbdc6 100644
--- a/CHANGELOG.rst
+++ b/CHANGELOG.rst
@@ -1,50 +1,17 @@
 Changelog
 =========
 
-1.0.0 (2022-12-14)
+0.1.0 (2023-03-10)
 -------------------
 
-* Initial release, with the following **6** workflow Operators/Parsers:
+Features
 
-.. list-table::
-   :header-rows: 1
+* **DatabricksWorkflowTaskGroup**: Airflow `Task Group <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#taskgroups>`_ that allows users to create a `Databricks Workflow <https://www.databricks.com/product/workflows>`_.
+* **DatabricksNotebookOperator**: Airflow `Operator <https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html>`_ which abstracts a pre-existing `Databricks Notebook <https://docs.databricks.com/notebooks/>`_. Can be used independently to run the Notebook, or within a Databricks Workflow Task Group.
+* **AstroDatabricksPlugin**: An Airflow `Plugin <https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html>`_ which is installed by the default. It allows users, by using the UI, to view a Databricks job and retry running it in case of failure.
 
-   * - Operator/Sensor Class
-     - Import Path
-     - Example DAG
+Known limitations
 
-   * - ``DBTTestOperator``
-     - .. code-block:: python
+* Cancelling the Airflow task does not stop its execution in Databricks (issue `#1 <https://github.com/astronomer/astro-providers-databricks/issues/1>`_).
+* Users should not click the buttons "repair all" or "repair single task" while the DAG/task is running (issue `#2 <https://github.com/astronomer/astro-providers-databricks/issues/2>`_).
 
-        from cosmos.providers.dbt.core.operators import DBTBaseOperator
-     - N/A
-
-   * - ``DBTSeedOperator``
-     - .. code-block:: python
-
-        from cosmos.providers.dbt.core.operators import DBTSeedOperator
-     - `Example DAG <https://github.com/astronomer/astronomer-cosmos/blob/1.0.0/examples/dags/extract_dag.py>`__
-
-   * - ``DBTRunOperator``
-     - .. code-block:: python
-
-        from cosmos.providers.dbt.core.operators import DBTRunOperator
-     - N/A
-
-   * - ``DBTTestOperator``
-     - .. code-block:: python
-
-        from cosmos.providers.dbt.core.operators import DBTTestOperator
-     - N/A
-
-   * - ``DbtDag``
-     - .. code-block:: python
-
-        from cosmos.providers.dbt.core.dag import DbtDag
-     - `Example DAG <https://github.com/astronomer/astronomer-cosmos/blob/1.0.0/examples/dags/attribution-playbook.py>`__
-
-   * - ``DbtTaskGroup``
-     - .. code-block:: python
-
-        from cosmos.providers.dbt.core.dag import DbtTaskGroup
-     - `Example DAG <https://github.com/astronomer/astronomer-cosmos/blob/1.0.0/examples/dags/jaffle_shop.py>`__
diff --git a/README.md b/README.md
index 24005ca..d290ab4 100644
--- a/README.md
+++ b/README.md
@@ -1,5 +1,105 @@
-# Astro Databricks
+<h1 align="center">
+  Astro Databricks
+</h1>
+  <h3 align="center">
+  Affordable Databricks Workflows in Apache Airflow<br><br>
+</h3>
 
-Astro Databricks is an airflow provider created by Astronomer for an optimal databricks experience.
+[![Python versions](https://img.shields.io/pypi/pyversions/astro-providers-databricks.svg)](https://pypi.org/pypi/astro-providers-databricks)
+[![License](https://img.shields.io/pypi/l/astro-providers-databricks.svg)](https://pypi.org/pypi/astro-providers-databricks)
+[![Development Status](https://img.shields.io/pypi/status/astro-providers-databricks.svg)](https://pypi.org/pypi/astro-providers-databricks)
+[![PyPI downloads](https://img.shields.io/pypi/dm/astro-providers-databricks.svg)](https://pypistats.org/packages/astro-providers-databricks)
+[![Contributors](https://img.shields.io/github/contributors/astronomer/astro-providers-databricks)](https://github.com/astronomer/astro-providers-databricks)
+[![Commit activity](https://img.shields.io/github/commit-activity/m/astronomer/astro-providers-databricks)](https://github.com/astronomer/astro-providers-databricks)
+[![CI](https://github.com/astronomer/astro-providers-databricks/actions/workflows/ci.yml/badge.svg)](https://github.com/astronomer/astro-providers-databricks)
+[![codecov](https://codecov.io/gh/astronomer/astro-providers-databricks/branch/main/graph/badge.svg?token=MI4SSE50Q6)](https://codecov.io/gh/astronomer/astro-providers-databricks)
 
-For up-to-date documentation, visit https://astronomer.github.io/astro-providers-databricks/
\ No newline at end of file
+
+**Astro Databricks** is an [Apache Airflow](https://github.com/apache/airflow) provider created by [Astronomer](https://www.astronomer.io/) for an **optimal Databricks experience**.  With the `DatabricksTaskGroup`, Astro Datatricks allows you to run from Databricks workflows without
+the need of running Jobs individually, which can result in [75% cost reduction](https://www.databricks.com/product/aws-pricing).
+
+## Prerequisites
+
+* Apache Airflow >= 2.2.4
+* Python >= 2.7
+* Databricks account
+* Previously created Databricks Notebooks
+
+## Install
+
+```shell
+pip install astro-providers-databricks
+```
+
+## Quickstart
+
+1. Use pre-existing or create two simple [Databricks Notebooks](https://docs.databricks.com/notebooks/). Their identifiers will be used in step (5). The original example DAG uses: 
+   * `Shared/Notebook_1`
+   * `Shared/Notebook_2`
+
+2. Generate a [Databricks Personal Token](https://docs.databricks.com/dev-tools/auth.html#databricks-personal-access-tokens). This will be used in step (6). 
+
+3. Ensure that your Airflow environment is set up correctly by running the following commands:
+
+    ```shell
+    export AIRFLOW_HOME=`pwd`
+   
+    airflow db init
+    ```
+   
+4. [Create using your preferred way](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html) a Databricks Airflow connection (so Airflow can access Databricks using your credentials). This can be done by running the following command, replacing the login and password (with your access token):
+
+```shell
+airflow connections add 'databricks_conn' \
+    --conn-json '{
+        "conn_type": "databricks",
+        "login": "some.email@yourcompany.com",
+        "host": "https://dbc-c9390870-65ef.cloud.databricks.com/",
+        "password": "personal-access-token"
+    }'
+```
+
+5. Copy the following workflow into a file named `example_databricks_workflow.py` and add it to the `dags` directory of your Airflow project:
+   
+   https://github.com/astronomer/astro-providers-databricks/blob/45897543a5e34d446c84b3fbc4f6f7a3ed16cdf7/example_dags/example_databricks_workflow.py#L48-L101
+
+   Alternatively, you can download `example_databricks_workflow.py`
+   ```shell
+    curl -O https://raw.githubusercontent.com/astronomer/astro-providers-databricks/main/example_dags/example_databricks_workflow.py
+   ```
+
+6. Run the example DAG:
+
+    ```sh
+    airflow dags test example_databricks_workflow `date -Iseconds`
+    ```
+   
+This will create a Databricks Workflow with two Notebook jobs.
+
+## Available features
+
+* `DatabricksWorkflowTaskGroup`: Airflow [task group](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/dags.html#taskgroups) that allows users to create a [Databricks Workflow](https://www.databricks.com/product/workflows).
+* `DatabricksNotebookOperator`: Airflow [operator](https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/operators.html) which abstracts a pre-existing [Databricks Notebook](https://docs.databricks.com/notebooks/). Can be used independently to run the Notebook, or within a Databricks Workflow Task Group.
+* `AstroDatabricksPlugin`: An Airflow [plugin](https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/plugins.html) which is installed by the default. It allows users, by using the UI, to view a Databricks job and retry running it in case of failure.
+
+## Documentation
+
+The documentation is a work in progress--we aim to follow the [Diátaxis](https://diataxis.fr/) system:
+
+* [Reference Guide](https://astronomer.github.io/astro-providers-databricks/)
+
+## Changelog
+
+Astro Databricks follows [semantic versioning](https://semver.org/) for releases. Read [changelog](CHANGELOG.rst) to understand more about the changes introduced to each version.
+
+## Contribution guidelines
+
+All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
+
+Read the [Contribution Guidelines](docs/contributing.rst) for a detailed overview on how to contribute.
+
+Contributors and maintainers should abide by the [Contributor Code of Conduct](CODE_OF_CONDUCT.md).
+
+## License
+
+[Apache Licence 2.0](LICENSE)
\ No newline at end of file
diff --git a/docs/contributing.rst b/docs/contributing.rst
index f6a332d..6b8ed76 100644
--- a/docs/contributing.rst
+++ b/docs/contributing.rst
@@ -1,17 +1,17 @@
-Cosmos Contributing Guide
+Contributing Guide
 =========================
 
 All contributions, bug reports, bug fixes, documentation improvements, enhancements are welcome.
 
 As contributors and maintainers to this project, you are expected to abide by the
-`Contributor Code of Conduct <https://github.com/astronomer/astronomer-cosmos/blob/main/CODE_OF_CONDUCT.md>`_.
+`Contributor Code of Conduct <https://github.com/astronomer/astro-providers-databricks/blob/main/CODE_OF_CONDUCT.md>`_.
 
 Overview
 ________
 
-To contribute to the cosmos project:
+To contribute to the **Astro Databricks** project:
 
-#. Please create a `GitHub Issue <https://github.com/astronomer/astronomer-cosmos/issues>`_ describing your contribution
+#. Please create a `GitHub Issue <https://github.com/astronomer/astro-providers-databricks/issues>`_ describing your contribution
 #. Open a feature branch off of the ``main`` branch and create a Pull Request into the ``main`` branch from your feature branch
 #. Link your issue to the pull request
 #. Once developments are complete on your feature branch, request a review and it will be merged once approved.
@@ -21,26 +21,30 @@ __________________________________
 
 Pre-requisites
 **************
-#. `Astro CLI <https://docs.astronomer.io/astro/cli/install-cli>`_
-#. `git <https://git-scm.com/book/en/v2/Getting-Started-Installing-Git>`_
 
-Local Sandbox
-************
-To create a sandbox where you can do real-time testing for your proposed to changes to Cosmos, see the corresponding
-development repository: `cosmos-dev <https://github.com/astronomer/cosmos-dev>`_.
+* pre-commmit
+* nox
+
+.. code-block:: bash
+    pip install nox pre-commit
+
+Create within `tests/` a `test-connections.yaml` file, similar to :download:`the CI's <../.github/ci-test-connections.yaml>`_.
+
+
+Run tests
+*****************
+
+.. code-block:: bash
+    nox -s "test-3.8(airflow='2.5')"
 
 Pre-Commit
 ************
 
 We use pre-commit to run a number of checks on the code before committing. To install pre-commit, run the following from
-your cloned ``astronomer-cosmos`` directory:
+your cloned ``astro-providers-databricks`` directory:
 
 .. code-block:: bash
 
-    python3 -m venv venv
-    source venv/bin/activate
-    pip install -r dev-requirements.txt
-    pip install pre-commit
     pre-commit install
 
 
diff --git a/tests/example_dags/.airflowignore b/example_dags/.airflowignore
similarity index 100%
rename from tests/example_dags/.airflowignore
rename to example_dags/.airflowignore
diff --git a/tests/example_dags/__init__.py b/example_dags/__init__.py
similarity index 100%
rename from tests/example_dags/__init__.py
rename to example_dags/__init__.py
diff --git a/tests/example_dags/example_databricks_notebook.py b/example_dags/example_databricks_notebook.py
similarity index 96%
rename from tests/example_dags/example_databricks_notebook.py
rename to example_dags/example_databricks_notebook.py
index 5ea84a8..2c812ef 100644
--- a/tests/example_dags/example_databricks_notebook.py
+++ b/example_dags/example_databricks_notebook.py
@@ -5,7 +5,7 @@
 from airflow.models.dag import DAG
 from airflow.utils.timezone import datetime
 
-from astro_databricks.operators.notebook import DatabricksNotebookOperator
+from astro_databricks import DatabricksNotebookOperator
 
 EXECUTION_TIMEOUT = int(os.getenv("EXECUTION_TIMEOUT", 6))
 default_args = {
diff --git a/tests/example_dags/example_databricks_workflow.py b/example_dags/example_databricks_workflow.py
similarity index 95%
rename from tests/example_dags/example_databricks_workflow.py
rename to example_dags/example_databricks_workflow.py
index ba3b993..9ec0dee 100644
--- a/tests/example_dags/example_databricks_workflow.py
+++ b/example_dags/example_databricks_workflow.py
@@ -5,8 +5,7 @@
 from airflow.models.dag import DAG
 from airflow.utils.timezone import datetime
 
-from astro_databricks.operators.notebook import DatabricksNotebookOperator
-from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
+from astro_databricks import DatabricksNotebookOperator, DatabricksWorkflowTaskGroup
 
 EXECUTION_TIMEOUT = int(os.getenv("EXECUTION_TIMEOUT", 6))
 default_args = {
diff --git a/pyproject.toml b/pyproject.toml
index f4594e5..06c74f4 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -2,11 +2,10 @@
 requires = ["hatchling"]
 build-backend = "hatchling.build"
 
-# TODO: change the name of the package
 [project]
 name = "astro-providers-databricks"
 dynamic = ["version"]
-description = "Astronomer SDK Template"
+description = "Affordable Databricks Workflows in Apache Airflow"
 readme = "README.md"
 license = "Apache-2.0"
 requires-python = ">=3.7"
@@ -64,8 +63,8 @@ tests = [
 
 # TODO: change the name of the package
 [project.urls]
-Homepage = "https://github.com/astronomer/astronomer-sdk-template"
-Documentation = "https://github.com/astronomer/astronomer-sdk-template"
+Homepage = "https://github.com/astronomer/astro-providers-databricks/"
+Documentation = "https://github.com/astronomer/astro-providers-databricks/"
 
 [tool.hatch.version]
 path = "src/astro_databricks/__init__.py"
@@ -108,7 +107,7 @@ fix = true
 
 [tool.pytest.ini_options]
 minversion = "6.0"
-addopts = "-s --cov=src --cov-branch --cov-report term-missing"
+addopts = "--cov=src --cov-branch --cov-report term-missing"
 testpaths = [
     "tests",
 ]
diff --git a/src/astro_databricks/__init__.py b/src/astro_databricks/__init__.py
index 225f59a..b62b998 100644
--- a/src/astro_databricks/__init__.py
+++ b/src/astro_databricks/__init__.py
@@ -1,3 +1,10 @@
 """Description of the package"""
+from astro_databricks.operators.notebook import DatabricksNotebookOperator
+from astro_databricks.operators.workflow import DatabricksWorkflowTaskGroup
 
 __version__ = "0.1.0"
+__all__ = [
+    "DatabricksNotebookOperator",
+    "DatabricksWorkflowTaskGroup"
+]
+


Test patch:
diff --git a/tests/test_example_dags.py b/tests/test_example_dags.py
index 7d7aa23..7b63ac4 100644
--- a/tests/test_example_dags.py
+++ b/tests/test_example_dags.py
@@ -23,7 +23,7 @@ def session():
 
 def get_dag_bag() -> DagBag:
     """Create a DagBag by adding the files that are not supported to .airflowignore"""
-    example_dags_dir = Path(__file__).parent / "example_dags"
+    example_dags_dir = Path(__file__).parent.parent / "example_dags"
 
     dag_bag = DagBag(example_dags_dir, include_examples=False)
     assert dag_bag.dags