Repository: great-expectations/great_expectations

Issue:
Host a great_expectations Docker image
**Problem**

At Earnest we prefer using container-based Operators on Airflow rather than using the "Python Operators". This is because we previously were using Luigi with Python "Tasks" and felt the pain of accumulating dependencies in a shared image and tying up ETL related code with the scheduler.

**Solution**

A Docker image for great_expectations could be created, using checkpoint as an entrypoint (related PR https://github.com/great-expectations/great_expectations/pull/1351)

On Airflow, such an image could be used with the KubernetesPodOperator. Here's some code (obviously not working!) hopefully it helps getting the idea across.

In Airflow:

```python
KubernetesPodOperator(
        task_id='ge_transactions',
        image='great-expectations/great_expectations:v1.0',
        name='ge-transactions',
        cmds=[
            'run, 'transactions',
            '--validation-operator-name', 'validation_operator_prod',
            '--batch-kwargs', ...
        ],
        ...
    )
# I think it's important to be able to pass the "validation operator name", 
# in dev I would like analysts/engineers to be able to do everything locally.
# In production I want the validations to be pulled/pushed from and to S3,
# with central site documentations built iteratively across DAGs owned by different  teams

# {{ ds }} is the execution_date, if the batch kwargs use queries that filter on `{{ ds }}` 
# you could create idempotent data QC.

# I assume everything could be passed from the CLI. This is to avoid having
# users create their own Docker image just to include the `YAML` configs
# or having to integrate using configmaps (I think cli args are more analyst friendly)
```
**YAML vs CLI args**


I think as part of this effort, we should discuss the best way to parameterize checkpoint runs.

It might make sense to support checkpoints being ran without a YAML file (purely through CLI args). 
That's because 1. I think most of the parameters  will have to be dynamic due to dev/prod differences and to inject environment related information (execution_date), 2. not sure those parameters will be big enough to justify using a file vs CLI args, 3. these yaml files would have to be included in customized Docker images built on top of the GE-provided image, or added through ConfigMaps or sidecar/init containers in Kubernetes. 

Both could be supported, but for YAML files, I think the cli should support injecting vars, possibly the same way DBT supports it (YAML dictionaries) https://docs.getdbt.com/docs/building-a-dbt-project/building-models/using-variables/#defining-variables-on-the-command-line

e.g.

```yml
validation_operator_name: validation_operator_${env}
batches:
  - batch_kwargs:
      query: SELECT * FROM db_${env}.transactions WHERE ds = ${ds}
      datasource: redshift_${env}
    expectation_suite_names:
      - suite_one
      - suite_two
```

Note that this is a very Airflow-centric example, but the advantage of providing a container would also be being less tied to a specific scheduling technology. I'm thinking [KubeFlow](https://www.kubeflow.org/docs/pipelines/sdk/build-component/) and Data QC integrated in ML pipelines for example (although I never used KubeFlow)


Discussion:
@Aylr @eugmandel I tried to document in detail the use case for the Docker image I was talking about in https://github.com/great-expectations/great_expectations/pull/1351, let me know :)
This  write up is wonderful!

Resolving patch:
diff --git a/docker/build.py b/docker/build.py
new file mode 100644
--- /dev/null
+++ b/docker/build.py
@@ -0,0 +1,113 @@
+# isort:block
+import logging
+import pathlib
+import sys
+from typing import List
+
+import click
+
+import docker
+from docker import APIClient
+
+PROJECT_ROOT = str(pathlib.Path(__file__).parent.parent.absolute())
+if PROJECT_ROOT not in sys.path:
+    sys.path.append(PROJECT_ROOT)
+
+import versioneer  # isort:skip
+
+
+logging.basicConfig(
+    format="[%(asctime)s] {%(filename)s:%(lineno)d} %(levelname)s - %(message)s",
+    level=logging.INFO,
+)
+
+logger = logging.getLogger(__name__)
+
+PYTHON_DOCKER_TAGS = [
+    "3.6-buster",
+    "3.7-buster",
+]
+
+DOCKER_REPOSITORY = "great-expectations/great_expectations"
+
+client = docker.from_env()
+python_docker_tags_opt = click.option(
+    "--python-docker-tags",
+    "-p",
+    multiple=True,
+    default=PYTHON_DOCKER_TAGS,
+    help="Build the image on top of these python images",
+    show_default=True,
+)
+great_expectations_version_opt = click.option(
+    "--ge-version", "-g", default=versioneer.get_version(), show_default=True
+)
+docker_repository_opt = click.option(
+    "--repository",
+    "-r",
+    default=DOCKER_REPOSITORY,
+    help="Target repository for the docker images",
+)
+
+
+def mk_image_tag(python_docker_tag: str, ge_version: str):
+    return f"python-{python_docker_tag}-ge-{ge_version}"
+
+
+@click.group()
+def cli():
+    pass
+
+
+@cli.command()
+@python_docker_tags_opt
+@great_expectations_version_opt
+@docker_repository_opt
+def build_images(python_docker_tags: List[str], ge_version: str, repository: str):
+    for python_docker_tag in python_docker_tags:
+        tag = mk_image_tag(python_docker_tag, ge_version)
+        image_name = f"{repository}:{tag}"
+
+        logger.info("Building image: " + image_name)
+
+        for line in client.api.build(
+            path=".",
+            dockerfile="docker/Dockerfile",
+            tag=image_name,
+            buildargs={"PYTHON_DOCKER_TAG": python_docker_tag},
+            rm=True,
+            decode=True,
+        ):
+            stripped = line.get("stream", "").strip()
+            if stripped:
+                logger.info(stripped)
+
+
+@cli.command()
+@python_docker_tags_opt
+@great_expectations_version_opt
+@docker_repository_opt
+def push_images(python_docker_tags: List[str], ge_version: str, repository: str):
+    for python_docker_tag in python_docker_tags:
+        tag = mk_image_tag(python_docker_tag, ge_version)
+
+        logger.info(f"Pushing image {repository}:{tag}")
+
+        for line in client.api.push(
+            repository=repository, tag=tag, stream=True, decode=True
+        ):
+            status = line.get("status", "").strip()
+            if status:
+                progress = line.get("progress", "").strip()
+                if progress:
+                    logger.info(f"{status}: {progress}")
+                else:
+                    logger.info(status)
+
+            error_detail = line.get("errorDetail", {}).get("message", "").strip()
+            if error_detail:
+                logger.error(error_detail)
+
+
+if __name__ == "__main__":
+    cli()
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,7 +24,15 @@
         "spark": ["pyspark>=2.3.2"],
         "sqlalchemy": ["sqlalchemy>=1.2"],
         "airflow": ["apache-airflow[s3]>=1.9.0", "boto3>=1.7.3"],
-        "gcp": ["google-cloud-storage>=1.28.0"],
+        "gcp": [
+            "google-cloud>=0.34.0",
+            "google-cloud-storage>=1.28.0",
+            "google-cloud-secret-manager>=1.0.0",
+            "pybigquery==0.4.15",
+        ],
+        "redshift": ["psycopg2>=2.8"],
+        "s3": ["boto3>=1.14"],
+        "snowflake": ["snowflake-sqlalchemy>=1.2"],
     },
     "packages": find_packages(exclude=["docs*", "tests*", "examples*"]),
     "entry_points": {


Test patch:
diff --git a/tests/test_packaging.py b/tests/test_packaging.py
--- a/tests/test_packaging.py
+++ b/tests/test_packaging.py
@@ -43,6 +43,11 @@ def test_requirements_files():
             [f'{line.name}{"".join(line.specs[0])}' for line in rp.parse(req)]
         )
 
+    with open(file_relative_path(__file__, "../requirements-dev-publish.txt")) as req:
+        requirements_dev_publish = set(
+            [f'{line.name}{"".join(line.specs[0])}' for line in rp.parse(req)]
+        )
+
     assert requirements <= requirements_dev
 
     assert requirements_dev_util.intersection(requirements_dev_spark) == set()
@@ -59,6 +64,9 @@ def test_requirements_files():
 
     assert requirements_dev_test.intersection(requirements_dev_build) == set()
 
+    assert requirements_dev_publish.intersection(requirements_dev_test) == set()
+    assert requirements_dev_publish.intersection(requirements_dev_build) == set()
+
     assert (
         requirements_dev
         - (
@@ -68,6 +76,7 @@ def test_requirements_files():
             | requirements_dev_spark
             | requirements_dev_test
             | requirements_dev_build
+            | requirements_dev_publish
         )
         == set()
     )