Repository: dbt-labs/dbt-core
Pull request: New Dockerfile
resolves - Partially resolves #4495

### Description
Updated dockerfile.  Images created should retain the previous dockerfile's functionality without requiring all the scripting and env var juggling.

### Checklist

- [x] I have signed the [CLA](https://docs.getdbt.com/docs/contributor-license-agreements)
- [x] I have run this code in development and it appears to resolve the stated issue
- [x] This PR includes tests, or tests are not required/relevant for this PR
- [x] I have updated the `CHANGELOG.md` and added information about my change


Problem statement:
[Feature] Update Dockerfile and docker image release process
### Is there an existing feature request for this?

- [X] I have searched the existing issues

### Describe the Feature

Our Dockerfile/image release process has been at least somewhat broken since we divided the codebase into `dbt-core` + `<dbt-adaptername>`.  This ticket encompasses the work required to create a new Dockerfile and implement a new image building process coupled to Github Actions.

### Describe alternatives you've considered

N/A

### Who will this benefit?

Anyone who uses Docker

### Are you interested in contributing this feature?

_No response_

### Anything else?

_No response_

Hints from issue discussion:
Duplicate of CORE-21 in Jira.
@iknox-fa Thanks for opening!

Here's a proposed build matrix, for when we should actually build/tag/push new images:
| image / tag   | `latest`                                                           | minor latest (e.g. `1.0.latest`)                              | specific version (e.g. `1.0.1rc1`)                                       |
|---------------|--------------------------------------------------------------------|---------------------------------------------------------------|--------------------------------------------------------------------------|
| dbt-core      | **Triggered by:** every final release of dbt-core, if latest*                        | every final release in this minor version family (e.g. 1.0.1) | every individual release (including prereleases)                                    |
| dbt-[adapter] | every final release of dbt-core or this adapter plugin, if latest* | every final release of dbt-core or this adapter plugin in this minor version family        | every individual release (including prereleases) of dbt-core or this adapter plugin |

*The "latest" version of dbt should, at any point in time, mean the latest patch release of the latest minor version. E.g. after v1.1.0 is released, a subsequent v1.0.3 patch would no longer update the overall `latest` tag. It should still update the `1.0.latest` tag.

This won't handle the case where an adapter plugin depends on another adapter plugin (e.g. `dbt-redshift` &rarr; `dbt-postgres` &rarr; `dbt-core`), the intermediary plugin has an update, and neither `dbt-core` nor `dbt-redshift` does. I think that's going to be an exceedingly rare case, and we can always push a one-off manually if we need to.

Definitely open to your thoughts here!
@kwigley had release process thoughts as well- could you add them here please, Kyle?

Commit message: Updated refs to be tag-based because bumpversion doesn't understand 'latest'

Patch to review:
diff --git a/.bumpversion.cfg b/.bumpversion.cfg
index b7f332c4c17..c60dcf84bba 100644
--- a/.bumpversion.cfg
+++ b/.bumpversion.cfg
@@ -36,4 +36,4 @@ first_value = 1
 
 [bumpversion:file:plugins/postgres/dbt/adapters/postgres/__version__.py]
 
-[bumpversion:file:docker/requirements/requirements.txt]
+[bumpversion:file:docker/Dockerfile]
diff --git a/Dockerfile.test b/Dockerfile.test
index b8d1fc36ea1..9c0afbcac08 100644
--- a/Dockerfile.test
+++ b/Dockerfile.test
@@ -1,3 +1,8 @@
+##
+#  This dockerfile is used for local development and adapter testing only.
+#  See `/docker` for a generic and production-ready docker file
+##
+
 FROM ubuntu:20.04
 
 ENV DEBIAN_FRONTEND noninteractive
diff --git a/docker-compose.yml b/docker-compose.yml
index e0a6ea9816a..deb51662bf6 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,3 +1,8 @@
+##
+#  This compose file is used for local development and adapter testing only.
+#  See `/docker` for a generic and production-ready docker file
+##
+
 version: "3.5"
 services:
   database:
diff --git a/docker/Dockerfile b/docker/Dockerfile
index e5aac737a76..040b3b4cd00 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -1,34 +1,127 @@
-ARG BASE_IMAGE=python:3.8-slim-bullseye
+##
+#  Generic dockerfile for dbt image building.  
+#  See README for operational details
+##
 
-FROM $BASE_IMAGE
-ARG BASE_REQUIREMENTS_SRC_PATH
-ARG WHEEL_REQUIREMENTS_SRC_PATH
-ARG DIST_PATH
+##
+# base image (abstract)
+##
+FROM python:3.9-slim-bullseye as base
 
+# Build args
+# N.B. The refs updated automagically every release via bumpversion
+ARG dbt_core_ref=dbt-core@v1.0.0
+ARG dbt_postgres_ref=dbt-core@v1.0.0
+ARG dbt_redshift_ref=dbt-redshift@v1.0.0
+ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
+ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
+ARG dbt_spark_ref=dbt-spark@v1.0.0
+ARG dbt_spark_version=all
+ARG dbt_third_party
+
+# System setup
 RUN apt-get update \
   && apt-get dist-upgrade -y \
   && apt-get install -y --no-install-recommends \
-  git \
-  ssh-client \
-  software-properties-common \
-  make \
-  build-essential \
-  ca-certificates \
-  libpq-dev \
+    git \
+    ssh-client \
+    software-properties-common \
+    make \
+    build-essential \
+    ca-certificates \
+    libpq-dev \
   && apt-get clean \
-  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-RUN echo BASE_REQUIREMENTS_SRC_PATH=$BASE_REQUIREMENTS_SRC_PATH
-RUN echo WHEEL_REQUIREMENTS_SRC_PATH=$WHEEL_REQUIREMENTS_SRC_PATH
-RUN echo DIST_PATH=$DIST_PATH
-COPY $BASE_REQUIREMENTS_SRC_PATH ./requirements.txt
-COPY $WHEEL_REQUIREMENTS_SRC_PATH ./wheel_requirements.txt
-COPY $DIST_PATH ./dist
-RUN pip install --upgrade pip setuptools
-RUN pip install --requirement ./requirements.txt
-RUN pip install --requirement ./wheel_requirements.txt
+  && rm -rf \
+    /var/lib/apt/lists/* \
+    /tmp/* \
+    /var/tmp/*
+
+# Env vars
 ENV PYTHONIOENCODING=utf-8
-ENV LANG C.UTF-8
-WORKDIR /usr/app
+ENV LANG=C.UTF-8
+
+# Update python
+RUN python -m pip install --upgrade pip setuptools --no-cache-dir
+
+# Set docker basics
+WORKDIR /usr/app/dbt/
 VOLUME /usr/app
 ENTRYPOINT ["dbt"]
+
+##
+# dbt-core
+##
+FROM base as dbt-core
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+##
+# dbt-postgres
+##
+FROM base as dbt-postgres
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+
+##
+# dbt-redshift
+##
+FROM base as dbt-redshift
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_redshift_ref}#egg=dbt-redshift" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+##
+# dbt-bigquery
+##
+FROM base as dbt-bigquery
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_bigquery_ref}#egg=dbt-bigquery" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+
+##
+# dbt-snowflake
+##
+FROM base as dbt-snowflake
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+
+##
+# dbt-spark
+##
+FROM base as dbt-spark
+RUN apt-get update \
+  && apt-get dist-upgrade -y \
+  && apt-get install -y --no-install-recommends \
+    python-dev \
+    libsasl2-dev \
+    gcc \
+    unixodbc-dev \
+  && apt-get clean \
+  && rm -rf \
+    /var/lib/apt/lists/* \
+    /tmp/* \
+    /var/tmp/*
+
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_spark_ref}#egg=dbt-spark[${dbt_spark_version}]" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+
+##
+# dbt-third-party
+##
+FROM base as dbt-third-party
+RUN python -m pip install "${dbt_third_party}" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+
+##
+# dbt-all
+##
+FROM base as dbt-all
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_redshift_ref}#egg=dbt-redshift" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_bigquery_ref}#egg=dbt-bigquery" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_snowflake_ref}#egg=dbt-snowflake" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_spark_ref}#egg=dbt-spark[${dbt_spark_version}]" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
diff --git a/docker/README.md b/docker/README.md
new file mode 100644
index 00000000000..5a25a03494a
--- /dev/null
+++ b/docker/README.md
@@ -0,0 +1,83 @@
+# Docker for dbt
+This docker file is suitable for building dbt Docker images locally or using with CI/CD to automate populating a container registry.
+
+
+## Building an image:
+This Dockerfile can create images for the following targets, each named after the database they support:
+* `dbt-core` _(no db-adapter support)_
+* `dbt-postgres`
+* `dbt-redshift`
+* `dbt-bigquery`
+* `dbt-snowflake`
+* `dbt-spark`
+* `dbt-third-party` _(requires additional build-arg)_
+* `dbt-all` _(installs all of the above in a single image)_
+
+In order to build a new image, run the following docker command.
+```
+docker build --tag <your_image_name>  --target <target_name> <path/to/dockerfile>
+```
+By default the images will be populated with the most recent release of `dbt-core` and whatever database adapter you select.  If you need to create an image from a different you can specify it by git ref using the `--build-arg` flag:
+```
+docker build --tag <your_image_name> \
+  --target <target_name> \
+  --build-arg <arg_name>=<git_ref> \
+  <path/to/dockerfile>
+```
+valid arg names for versioning are:
+* `dbt_core_ref`
+* `dbt_postgres_ref`
+* `dbt_redshift_ref`
+* `dbt_bigquery_ref`
+* `dbt_snowflake_ref`
+* `dbt_spark_ref`
+
+If you wish to build an image with a third-party adapter you can use the `dbt-third-party` target.  This target requires you provide a path to the adapter that can be processed by `pip` by using the `dbt_third_party` build arg:
+```
+docker build --tag <your_image_name> \
+  --target dbt-third-party \ 
+  --build-arg dbt_third_party=<pip_parsable_install_string> \ 
+  <path/to/dockerfile>
+```
+
+> Note: By overiding build args it is very possible to build an image with incompatible versions of adapter and core!  
+>
+>TODO: Provide compatibility matrix or other instructions to determine corectness of versions.
+
+### Examples:
+To build an image named "my-dbt" that supports redshift using the latest releases:
+```
+cd dbt-core/docker
+docker build --tag my-dbt  --target dbt-redshift .
+```
+
+To build an image named "my-other-dbt" that supports bigquery using `dbt-core` version 0.21.latest and the bigquery adapter version 1.0.0b1:
+```
+cd dbt-core/docker
+docker build \
+  --tag my-other-dbt  \
+  --target dbt-bigquery \
+  --build-arg dbt_bigquery_ref=dbt-bigquery@v1.0.0b1 \
+  --build-arg dbt_core_ref=dbt-core@0.21.latest  \
+ .
+```
+
+To build an image named "my-third-party-dbt" that uses [Materilize third party adapter](https://github.com/MaterializeInc/materialize/tree/main/misc/dbt-materialize) and the latest release of `dbt-core`:
+```
+cd dbt-core/docker
+docker build --tag my-third-party-dbt \
+  --target dbt-third-party \
+  --build-arg dbt_third_party=dbt-materialize \
+  .
+```
+
+## Special cases
+There are a few special cases worth noting:
+* The `dbt-spark` database adapter comes in three different versions named `PyHive`, `ODBC`, and the default `all`.  If you wish to overide this you can use the `--build-arg` flag with the value of `dbt_spark_version=<version_name>`.  See the [docs](https://docs.getdbt.com/reference/warehouse-profiles/spark-profile) for more information.
+
+## Running an image in a container:
+The `ENTRYPOINT` for this Dockerfile is the command `dbt` so you can bind-mount your project to `/usr/app` and use dbt as normal:
+```
+docker run --mount type=bind,source=path/to/project,target=/usr/app my-dbt ls
+```
+> Note: bind-mount sources _must_ be an absolute path
diff --git a/docker/requirements/requirements.txt b/docker/requirements/requirements.txt
deleted file mode 100644
index bf1a134a4eb..00000000000
--- a/docker/requirements/requirements.txt
+++ /dev/null
@@ -1,44 +0,0 @@
-agate==1.6.3
-attrs==21.2.0
-Babel==2.9.1
-certifi==2021.10.8
-cffi==1.15.0
-charset-normalizer==2.0.8
-click==8.0.3
-colorama==0.4.4
-dbt-core==1.0.0
-dbt-extractor==0.4.0
-dbt-postgres==1.0.0
-future==0.18.2
-hologram==0.0.14
-idna==3.3
-importlib-metadata==4.8.2
-isodate==0.6.0
-Jinja2==2.11.3
-jsonschema==3.1.1
-leather==0.3.4
-Logbook==1.5.3
-MarkupSafe==2.0.1
-mashumaro==2.9
-minimal-snowplow-tracker==0.0.2
-msgpack==1.0.3
-networkx==2.6.3
-packaging==21.3
-parsedatetime==2.4
-psycopg2-binary==2.9.2
-pycparser==2.21
-pyparsing==3.0.6
-pyrsistent==0.18.0
-python-dateutil==2.8.2
-python-slugify==5.0.2
-pytimeparse==1.1.8
-pytz==2021.3
-PyYAML==6.0
-requests==2.26.0
-six==1.16.0
-sqlparse==0.4.2
-text-unidecode==1.3
-typing-extensions==3.10.0.2
-urllib3==1.26.7
-Werkzeug==2.0.2
-zipp==3.6.0


Code review comments:
diff_hunk: @@ -1,34 +1,127 @@
-ARG BASE_IMAGE=python:3.8-slim-bullseye
+##
+#  Generic dockerfile for dbt image building.  
+#  See README for operational details
+##
 
-FROM $BASE_IMAGE
-ARG BASE_REQUIREMENTS_SRC_PATH
-ARG WHEEL_REQUIREMENTS_SRC_PATH
-ARG DIST_PATH
+##
+# base image (abstract)
+##
+FROM python:3.9-slim-bullseye as base
 
+# Build args
+# N.B. The refs updated automagically every release via bumpversion
+ARG dbt_core_ref=dbt-core@v1.0.0
+ARG dbt_postgres_ref=dbt-core@v1.0.0
+ARG dbt_redshift_ref=dbt-redshift@v1.0.0
+ARG dbt_bigquery_ref=dbt-bigquery@v1.0.0
+ARG dbt_snowflake_ref=dbt-snowflake@v1.0.0
+ARG dbt_spark_ref=dbt-spark@v1.0.0
+ARG dbt_spark_version=all
+ARG dbt_third_party
+
+# System setup
 RUN apt-get update \
   && apt-get dist-upgrade -y \
   && apt-get install -y --no-install-recommends \
-  git \
-  ssh-client \
-  software-properties-common \
-  make \
-  build-essential \
-  ca-certificates \
-  libpq-dev \
+    git \
+    ssh-client \
+    software-properties-common \
+    make \
+    build-essential \
+    ca-certificates \
+    libpq-dev \
   && apt-get clean \
-  && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
-
-RUN echo BASE_REQUIREMENTS_SRC_PATH=$BASE_REQUIREMENTS_SRC_PATH
-RUN echo WHEEL_REQUIREMENTS_SRC_PATH=$WHEEL_REQUIREMENTS_SRC_PATH
-RUN echo DIST_PATH=$DIST_PATH
-COPY $BASE_REQUIREMENTS_SRC_PATH ./requirements.txt
-COPY $WHEEL_REQUIREMENTS_SRC_PATH ./wheel_requirements.txt
-COPY $DIST_PATH ./dist
-RUN pip install --upgrade pip setuptools
-RUN pip install --requirement ./requirements.txt
-RUN pip install --requirement ./wheel_requirements.txt
+  && rm -rf \
+    /var/lib/apt/lists/* \
+    /tmp/* \
+    /var/tmp/*
+
+# Env vars
 ENV PYTHONIOENCODING=utf-8
-ENV LANG C.UTF-8
-WORKDIR /usr/app
+ENV LANG=C.UTF-8
+
+# Update python
+RUN python -m pip install --upgrade pip setuptools --no-cache-dir
+
+# Set docker basics
+WORKDIR /usr/app/dbt/
 VOLUME /usr/app
 ENTRYPOINT ["dbt"]
+
+##
+# dbt-core
+##
+FROM base as dbt-core
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
+
+##
+# dbt-postgres
+##
+FROM base as dbt-postgres
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres" --no-cache-dir
+RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_core_ref}#egg=dbt-core&subdirectory=core" --no-cache-dir
path: docker/Dockerfile
text: @user1:
🌶️ take, I don't think we need to worry about the specifying the dbt-core version for each database adapter specific installation. It creates quite the matrix to manage. I can be easily convinced that my idea is not a good one though.

```suggestion
FROM base as dbt-postgres
RUN python -m pip install "git+https://github.com/dbt-labs/${dbt_postgres_ref}#egg=dbt-postgres&subdirectory=plugins/postgres" --no-cache-dir
```

I'm unsure how the order of the pip install lines with the `--no-cache-dir` work in practice as well. The command to install dbt-postgres will use pip to resolve which compatible version of dbt-core to install and all of its dependencies. The command to install dbt-core will install dbt-core with the version using the ref specified and all of its dependencies again. I think combining this on one line will allow pip to install everything and resolve dependency versions once.

@author:
> 🌶️ take, I don't think we need to worry about the specifying the dbt-core version for each database adapter specific installation

I completely agree.  That functionality was added because I can see it being useful as a debugging tool, we don't have to build/push images for every combination (but we could).

@author:
> I'm unsure how the order of the pip install lines with the --no-cache-dir work in practice as well. The command to install dbt-postgres will use pip to resolve which compatible version of dbt-core to install and all of its dependencies. The command to install dbt-core will install dbt-core with the version using the ref specified and all of its dependencies again. I think combining this on one line will allow pip to install everything and resolve dependency versions once.

So there's sort of two things to unpack here:

1:  Do we want to support mix-n-matching db adapter + core versions
  My take:  Yes, if it's possible to do without causing problems for builds that don't mix versions, 
  although I'm happy to re-asses that.

2:  Multiple run commands require multiple dependency resolutions, which is inefficient.
  My take:  This is a good point-- I should be able to combine them into a single RUN.

@user1:
1. sounds good to me!
2. I think I more meant that I'm not sure if `pip install dbt-core dbt-postgres` and `pip install dbt-core && dbt-postgres` will result in the same thing. Maybe it is! I haven't dug into pip in some time.

@author:
Oh I see what you're saying... I'll do some playing around and see if that has the same result.  I still like the idea of putting them in the same RUN layer though.