Repository: dbt-labs/dbt-core

Issue:
Use psycopg2-binary dep for Postgres and Redshift when v2.8 comes out
The `psycopg2` dep currently ships as a wheel, but version 2.8 will revert psycopg2 back to being source-only. This won't be a problem for dbt, as we've set the upper bound of psycopg2 to 2.8.

When the 2.8 release eventually comes out, we should upgrade and switch dbt to use the psycopg2-binary dep instead.

Information about releases [here](http://initd.org/psycopg/).
And info about the binary release [here](http://initd.org/psycopg/articles/2018/02/08/psycopg-274-released/).


Discussion:
Tentatively including this in the 0.14.1 milestone, but it may make sense to push this to a minor release since the upgrade path might be non-trivial.

It sounds like we'll need to run uninstall `psycopg2` and then install `psycopg2-binary` -- does that mean that installation with `pip` will require the `-I` flag for upgrading users?

Last: it sounds like `psycopg2-binary` isn't intended for use in libraries. Let's figure out if using the `-binary` wheel is indeed a good idea for us to use here, and if so, what (if any) caveats exist.
I've done far more reading on this than I'd like, and what I think I want to do is require `psycopg2-binary` by default on all platforms, but allow users to override their install to use the non-binary form.

My reasoning is that on macos and windows especially, building psycopg2 locally requires a bunch of headers that the average user is realistically just never going to install. Since so far the wheels have been working fine for us, I don't think it's very reasonable to change that and require you to have a compiler toolchain + libpq headers.

On the other hand, on Linux it's apparently reasonably likely to get mystery segfaults when openssl versions change or whatever. I've never experienced that on our docker images but I'm willing to believe it - this poor guy wouldn't have gone through all this nonsense unless he had a compelling reason! So we probably need to at least provide some way to let those users get psycopg2 installed.

I'm not sure exactly how to do this but I do know that setup.py provides a few ways to pass flags along. The real painful part is that we have to pass those arguments through to our plugins, if provided. I think the easiest thing to do is probably to just use environment variables to decide if we should dynamically set the `install_requires` used in `setup()`. Something like this in `plugins/postgres/setup.py`:

```python
install_requires=[
   '{} >=2.8.2, <2.9'.format(os.getenv('DBT_PSYCOPG2_PACKAGE', 'psycopg2-binary')),
  ...
]
```

Then if you know better, you can always set `DBT_PSYCOPG2_PACKAGE="psycopg2"` before you `pip install`.

There are two big downsides here:

   1) The upgrade path is awful. This is pip's fault, basically - it provides no meaningful facility to note that two pypi packages provide the same package. Maybe we can do some install-time shenanigans to decide if psycopg2 already exists, and if so check its version and either uninstall it (if it's `psycopg2`) or upgrade it (if it's `psychopg2-binary`). I'm sure pip folks will hate that, but they made this mess so I have no sympathy.

  2) If you have another package in your same virtualenv that requires the opposite package as the one you've selected, things will get very weird! This is fundamentally related to problem 1, but it's more of a user-level issue. Caveat emptor, I guess? Use a fresh virtualenv for dbt or make sure you set the environment variable to perfectly match the existing install!

The way I see it we don't have much choice but to default to `psycopg2-binary` - getting your average mac/windows user to install libpq development headers and a compiler toolchain is not a realistic option, and staying on 2.7.x obviously isn't an option on Windows given the concurrency issues w/OpenSSL 1.0.
@beckjake do you think we should do this for 0.14.1? Or should we hold off for 0.15.0?
I feel pretty comfortable with it on 0.14.1 if we feel good about the pain of changing dependencies on a minor release. I've been using it locally since this issue got written and everything seems fine.
Ok - I recall you mentioning that `pip` might do something unfortunate if users try to `--upgrade` their dbt version. Am I misremembering, or do you think special care will need to be taken when upgrading to 0.14.1 from a previous version?
We can test it out, but I would guess upgrades end in users who have both the old `psycopg2` and the new `psycopg2-binary`, and a race condition over which path wins during import.
@beckjake I think I want to kick this out of the 0.14.1 milestone. I think your ideas here are good ones, but I have so little faith in pip doing the right for us thing here, and it would be really unfortunate if we bricked a lot of dbt installations in weird ways in a bugfix/patch release.

Let's re-prioritize this for 0.15.0 -- that should give us some time to explore (and test) the approaches you've outlined in your comment above.

Resolving patch:
diff --git a/plugins/postgres/setup.py b/plugins/postgres/setup.py
--- a/plugins/postgres/setup.py
+++ b/plugins/postgres/setup.py
@@ -2,6 +2,24 @@
 from setuptools import find_namespace_packages
 from setuptools import setup
 import os
+import sys
+
+
+def _dbt_psycopg2_name():
+    # if the user chose something, use that
+    package_name = os.getenv('DBT_PSYCOPG2_NAME', '')
+    if package_name:
+        return package_name
+
+    binary_only_versions = [(3, 8)]
+
+    # binary wheels don't exist for all versions. Require psycopg2-binary for
+    # them and wait for psycopg2.
+    if sys.version_info[:2] in binary_only_versions:
+        return 'psycopg2-binary'
+    else:
+        return 'psycopg2'
+
 
 package_name = "dbt-postgres"
 package_version = "0.15.0b2"
@@ -11,6 +29,8 @@
 with open(os.path.join(this_directory, 'README.md')) as f:
     long_description = f.read()
 
+DBT_PSYCOPG2_NAME = _dbt_psycopg2_name()
+
 setup(
     name=package_name,
     version=package_version,
@@ -30,7 +50,7 @@
     },
     install_requires=[
         'dbt-core=={}'.format(package_version),
-        'psycopg2>=2.8.4,<2.9',
+        '{}~=2.8'.format(DBT_PSYCOPG2_NAME),
     ],
     zip_safe=False,
     classifiers=[
diff --git a/plugins/redshift/setup.py b/plugins/redshift/setup.py
--- a/plugins/redshift/setup.py
+++ b/plugins/redshift/setup.py
@@ -3,6 +3,7 @@
 from setuptools import setup
 import os
 
+
 package_name = "dbt-redshift"
 package_version = "0.15.0b2"
 description = """The redshift adapter plugin for dbt (data build tool)"""
@@ -33,7 +34,6 @@
         'dbt-postgres=={}'.format(package_version),
         'boto3>=1.6.23,<1.10.0',
         'botocore>=1.9.23,<1.13.0',
-        'psycopg2>=2.7.5,<2.8',
     ],
     zip_safe=False,
     classifiers=[


Test patch: