Repository: dbt-labs/dbt-core

Issue:
[CT-326] [Bug] incompatible build for `dbt-core`
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

It is currently not possible to install `dbt-core` because of incompatible versions of `google-api-core`.

```
ERROR: Could not find a version that matches google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<1.31.3,<2.0.0dev,<3.0.0dev,>=1.16.0,>=1.21.0,>=1.31.5 (from dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
Tried: 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.1.3, 0.1.4, 0.1.4, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.11.1, 1.11.1, 1.12.0, 1.12.0, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.14.1, 1.14.1, 1.14.2, 1.14.2, 1.14.3, 1.14.3, 1.15.0, 1.15.0, 1.16.0, 1.16.0, 1.17.0, 1.17.0, 1.18.0, 1.18.0, 1.19.0, 1.19.0, 1.19.1, 1.19.1, 1.20.0, 1.20.0, 1.20.1, 1.20.1, 1.21.0, 1.21.0, 1.22.0, 1.22.0, 1.22.1, 1.22.1, 1.22.2, 1.22.2, 1.22.3, 1.22.3, 1.22.4, 1.22.4, 1.23.0, 1.23.0, 1.24.0, 1.24.0, 1.24.1, 1.24.1, 1.25.0, 1.25.0, 1.25.1, 1.25.1, 1.26.0, 1.26.0, 1.26.1, 1.26.1, 1.26.2, 1.26.2, 1.26.3, 1.26.3, 1.27.0, 1.27.0, 1.28.0, 1.28.0, 1.29.0, 1.29.0, 1.30.0, 1.30.0, 1.31.0, 1.31.0, 1.31.1, 1.31.1, 1.31.2, 1.31.2, 1.31.3, 1.31.3, 1.31.4, 1.31.4, 1.31.5, 1.31.5, 2.0.0, 2.0.0, 2.0.1, 2.0.1, 2.1.0, 2.1.0, 2.1.1, 2.1.1, 2.2.0, 2.2.0, 2.2.1, 2.2.1, 2.2.2, 2.2.2, 2.3.0, 2.3.0, 2.3.1, 2.3.1, 2.3.2, 2.3.2, 2.4.0, 2.4.0, 2.5.0, 2.5.0, 2.6.0, 2.6.0
Skipped pre-versions: 1.6.0a1, 1.6.0a1, 1.26.0.dev0, 1.26.0.dev0, 2.0.0b1, 2.0.0b1
There are incompatible versions in the resolved dependencies:
  google-api-core<1.31.3,>=1.16.0 (from dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
  google-api-core<2.0.0dev,>=1.21.0 (from google-cloud-core==1.7.2->dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
  google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 (from google-cloud-bigquery==2.34.2->dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
```

It is also not possible to install `dbt~=0.21` through Pipenv because of incompatible versions of `typing-extensions` required by `mashumaro==2.9` and `typing-extensions>=3.7.4,<3.11` defined in [setup.py](https://github.com/dbt-labs/dbt-core/blob/main/core/setup.py).

```
ERROR: Could not find a version that matches typing-extensions<3.11,>=3.7.4,>=4.0.1 (from dbt-core==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvw3y63obzrequirements/pipenv-iw7fdxvz-constraints.txt (line 2))
Tried: 3.6.2, 3.6.2, 3.6.2.1, 3.6.2.1, 3.6.5, 3.6.5, 3.6.6, 3.6.6, 3.7.2, 3.7.2, 3.7.4, 3.7.4, 3.7.4.1, 3.7.4.1, 3.7.4.2, 3.7.4.2, 3.7.4.3, 3.7.4.3, 3.10.0.0, 3.10.0.0, 3.10.0.1, 3.10.0.1, 3.10.0.2, 3.10.0.2, 4.0.0, 4.0.0, 4.0.1, 4.0.1, 4.1.0, 4.1.0, 4.1.1, 4.1.1
There are incompatible versions in the resolved dependencies:
  typing-extensions (from mashumaro==2.5->dbt-core==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvw3y63obzrequirements/pipenv-iw7fdxvz-constraints.txt (line 2))
  typing-extensions<3.11,>=3.7.4 (from dbt-core==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvw3y63obzrequirements/pipenv-iw7fdxvz-constraints.txt (line 2))
  typing-extensions>=4.0.1 (from azure-core==1.23.0->azure-storage-blob==12.9.0->snowflake-connector-python[secure-local-storage]==2.5.1->dbt-snowflake==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvw3y63obzrequirements/pipenv-iw7fdxvz-constraints.txt (line 2))
```

This is not the first time that `dbt` has defined incompatible versions in its dependencies (see #4171).

Currently, `dbt-core` pins specific versions of dependencies defined in [setup.py](https://github.com/dbt-labs/dbt-core/blob/main/core/setup.py) which is overly restrictive and prevents other packages from being installed in the same environment (and as is the case now, even just `dbt` alone in the environment).

For example, here are the currently defined dependencies:

```python
    install_requires=[
        "Jinja2==2.11.3",
        "MarkupSafe==2.0.1",
        "agate>=1.6,<1.6.4",
        "click>=7.0,<9",
        "colorama>=0.3.9,<0.4.5",
        "hologram==0.0.14",
        "isodate>=0.6,<0.7",
        "logbook>=1.5,<1.6",
        "mashumaro==2.9",
        "minimal-snowplow-tracker==0.0.2",
        "networkx>=2.3,<3",
        "packaging>=20.9,<22.0",
        "sqlparse>=0.2.3,<0.5",
        "dbt-extractor==0.4.0",
        "typing-extensions>=3.7.4,<3.11",
        "werkzeug>=1,<3",
        # the following are all to match snowflake-connector-python
        "requests<3.0.0",
        "idna>=2.5,<4",
        "cffi>=1.9,<2.0.0",
    ]
```

According to the [Python Packaging User Guide](https://packaging.python.org/en/latest/) maintained by the Python Packaging Authority (PyPA):

> It is not considered best practice to use `install_requires` to pin dependencies to specific versions, or to specify sub-dependencies (i.e. dependencies of your dependencies). This is overly-restrictive, and prevents the user from gaining the benefit of dependency upgrades.

Source: [install-requires-vs-requirements](https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/)

### Expected Behavior

`dbt-core` should use non-conflicting versions of dependencies and sub-dependencies. Ideally, `dbt-core` should update the way it defines dependencies by not pinning specific versions. Version restrictions in `setup.py` should be reserved to known bugs or incompatibilities for specific packages. Pinning version numbers should be reserved for `requirements.txt` as recommended by [PyPA](https://packaging.python.org/en/latest/discussions/install-requires-vs-requirements/).

### Steps To Reproduce

1. `pip install pipenv`
2. `pipenv install dbt-core`

### Relevant log output

```shell
ERROR: Could not find a version that matches google-api-core!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<1.31.3,<2.0.0dev,<3.0.0dev,>=1.16.0,>=1.21.0,>=1.31.5 (from dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
Tried: 0.1.0, 0.1.0, 0.1.1, 0.1.1, 0.1.2, 0.1.2, 0.1.3, 0.1.3, 0.1.4, 0.1.4, 1.0.0, 1.0.0, 1.1.0, 1.1.0, 1.1.1, 1.1.1, 1.1.2, 1.1.2, 1.2.0, 1.2.0, 1.2.1, 1.2.1, 1.3.0, 1.3.0, 1.4.0, 1.4.0, 1.4.1, 1.4.1, 1.5.0, 1.5.0, 1.5.1, 1.5.1, 1.5.2, 1.5.2, 1.6.0, 1.6.0, 1.7.0, 1.7.0, 1.8.0, 1.8.0, 1.8.1, 1.8.1, 1.8.2, 1.8.2, 1.9.0, 1.9.0, 1.10.0, 1.10.0, 1.11.0, 1.11.0, 1.11.1, 1.11.1, 1.12.0, 1.12.0, 1.13.0, 1.13.0, 1.14.0, 1.14.0, 1.14.1, 1.14.1, 1.14.2, 1.14.2, 1.14.3, 1.14.3, 1.15.0, 1.15.0, 1.16.0, 1.16.0, 1.17.0, 1.17.0, 1.18.0, 1.18.0, 1.19.0, 1.19.0, 1.19.1, 1.19.1, 1.20.0, 1.20.0, 1.20.1, 1.20.1, 1.21.0, 1.21.0, 1.22.0, 1.22.0, 1.22.1, 1.22.1, 1.22.2, 1.22.2, 1.22.3, 1.22.3, 1.22.4, 1.22.4, 1.23.0, 1.23.0, 1.24.0, 1.24.0, 1.24.1, 1.24.1, 1.25.0, 1.25.0, 1.25.1, 1.25.1, 1.26.0, 1.26.0, 1.26.1, 1.26.1, 1.26.2, 1.26.2, 1.26.3, 1.26.3, 1.27.0, 1.27.0, 1.28.0, 1.28.0, 1.29.0, 1.29.0, 1.30.0, 1.30.0, 1.31.0, 1.31.0, 1.31.1, 1.31.1, 1.31.2, 1.31.2, 1.31.3, 1.31.3, 1.31.4, 1.31.4, 1.31.5, 1.31.5, 2.0.0, 2.0.0, 2.0.1, 2.0.1, 2.1.0, 2.1.0, 2.1.1, 2.1.1, 2.2.0, 2.2.0, 2.2.1, 2.2.1, 2.2.2, 2.2.2, 2.3.0, 2.3.0, 2.3.1, 2.3.1, 2.3.2, 2.3.2, 2.4.0, 2.4.0, 2.5.0, 2.5.0, 2.6.0, 2.6.0
Skipped pre-versions: 1.6.0a1, 1.6.0a1, 1.26.0.dev0, 1.26.0.dev0, 2.0.0b1, 2.0.0b1
There are incompatible versions in the resolved dependencies:
  google-api-core<1.31.3,>=1.16.0 (from dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
  google-api-core<2.0.0dev,>=1.21.0 (from google-cloud-core==1.7.2->dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
  google-api-core[grpc]!=2.0.*,!=2.1.*,!=2.2.*,!=2.3.0,<3.0.0dev,>=1.31.5 (from google-cloud-bigquery==2.34.2->dbt-bigquery==0.21.1->dbt==0.21.1->-r /var/folders/rq/bvjd2wms54j2m16rt180h5th0000gp/T/pipenvshohdu0arequirements/pipenv-hrvpe_f3-constraints.txt (line 2))
```
```


### Environment

```markdown
- OS: macos
- Python: 3.8
- dbt: 1.0
```


### What database are you using dbt with?

_No response_

### Additional Context

_No response_


Discussion:
I've got the same issue with `azure-core` and `typing-extensions` in a dagster+dbt project.
Same thing on my end for dagster+dbt! I also have another project with dbt+sqlfluff that has conflicts with `typing-extensions`
Hey @chasleslr , thanks for opening this issue! I agree with the idea that requirements in `install_requires` shouldn't be overly restrictive. And we should always have a good reason for pinning a dependency to an older major version.  Looks like the restriction for `typing-extensions` is too tight and we didn't have to do it. We have a bump for that at #4719 that we are going to get into the main branch and backported as needed. We will be actively looking into updating it so user can have better installation experience.
A few other things I would like to touch on while we are here.
* Looks like you are also trying to make something work with v0.21, with confusion among google dependency libraries. The split-up of adapter plugins makes installing each one easier — now, users won't ever have to manage shared dependencies of both google-api-core and snowflake-connector-python
* Starting in v1.0, we'll support each minor version for a full year, and backport fixes needed for installation/security. (We didn't used to do this for older versions, and we aren't going to do it for v0.21.) New docs on this: https://docs.getdbt.com/docs/core-versions

@ChenyuLInx I'm a Dagster dev and was just trying to add `typing-extensions` as a dep for us and ran into this problem. I can see the PR you linked has been merged to `main`, 10 days ago, but the latest release is 2 days ago and doesn't contain the bump-- is a release with the bump expected to come out soon?

Also, it looks like `typing-extensions` has still been capped at a minor version even in the PR. Do you mind my asking why? Seems like it's just going to cause the same problem down the line.

For anyone hoping to better understand when version capping is and is not a good idea, here's a _very_ thorough article on the matter: https://iscinumpy.dev/post/bound-version-constraints/

Resolving patch:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -65,7 +65,7 @@
         "packaging>=20.9,<22.0",
         "sqlparse>=0.2.3,<0.5",
         "dbt-extractor~=0.4.1",
-        "typing-extensions>=3.7.4,<4.2",
+        "typing-extensions>=3.7.4",
         "werkzeug>=1,<3",
         # the following are all to match snowflake-connector-python
         "requests<3.0.0",


Test patch: