Repository: dbt-labs/dbt-core

Issue:
[CT-1688] [Feature] Remove packaging <22 version cap
### Is this your first time submitting a feature request?

- [X] I have read the [expectations for open source contributors](https://docs.getdbt.com/docs/contributing/oss-expectations)
- [X] I have searched the existing issues, and I could not find an existing issue for this feature
- [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

### Describe the feature

This isn't really a "feature", but no other category fit. Please consider lifting `dbt-core`'s current upper version bound of `packaging<22`, it prevents installation in the same environment as tox 4, and I doubt the cap actually needs to be there-- just delete any references to `packaging.LegacyVersion` (which used to be a possible return from `packaging.version.parse`) and you can probably remove it.

### Describe alternatives you've considered

_No response_

### Who will this benefit?

Anyone working with `dbt-core` in an environment with a lot of other packages, and in particular in a dev environment with tox 4.

### Are you interested in contributing this feature?

I'd be happy to put up a PR, I grepped dbt-core codebase and packaging only used in a few places

### Anything else?

_No response_


Discussion:
Thanks for noticing this @smackesey !

We'd welcome your contribution on this. [Here's](https://github.com/dbt-labs/dbt-core/pull/5218/files) a similar PR you can use as an example. Obviously, your PR might be a little more involved depending on what you find.

Feel free to let me know if you run into any issues with the [contributing instructions](https://github.com/dbt-labs/dbt-core/blob/main/CONTRIBUTING.md#contributing-to-dbt-core).
Thanks @dbeatty10, as you can see I've opened a PR. I'd also like to make a more general point, which is that dbt (both core and other packages) are very aggressive with version capping-- it looks like caps are put on almost all dependencies as a matter of course, rather than only when specifically required.

Every version cap raises the probability of a versioning conflict with other libraries. Dbt's caps have caused us build pains over at [Dagster](https://github.com/dagster-io/dagster) a few times-- see [here](https://github.com/dbt-labs/dbt-core/issues/4833) for an older similar issue. In fact there is another issue even now, which is that the [dbt-snowflake](snowflake-connector-python) cap on `snowflake-connector-python` is conflicting with some of our packages (not sure how exactly, but it's causing build failures).

Dbt is a library likely to be used with all manner of other python packages, and dbt's dependency version caps  are probably causing your users more pain than you realize. As a fellow library developer in the data engineering space, I'd like to humbly request you reconsider your version capping philosophy, to a "cap only when needed" approach. If a new version of a library comes out that breaks dbt, your CI should catch it immediately and you can introduce a cap at that time.

A gentleman by the name of Henry Schreiner wrote something approximating a [small novel](https://iscinumpy.dev/post/bound-version-constraints/) on the subject of version caps in python libs-- it's a good read (even just the intro) for anyone trying to really understand this issue. Posting a TLDR excerpt below. Thanks.

>Capping dependencies has long term negative effects, especially for libraries, and should never be taken lightly. A library is not installed in isolation; it has to live with other libraries in a shared environment. Only add a cap if a dependency is known to be incompatible or there is a high (>75%) chance of it being incompatible in its next release. Do not cap by default - capping dependencies makes your software incompatible with other libraries that also have strict lower limits on dependencies, and limits future fixes. Anyone can fix a missing cap, but users cannot fix an over restrictive cap causing solver errors. It also encourages hiding issues until they become harder to fix, it does not scale to larger systems, it limits your ability to access security and bugfix updates, and some tools (Poetry) force these bad decisions on your downstream users if you make them. Never cap Python, it is fundamentally broken at the moment. Also, even packing capping has negative consequences that can produce unexpected solves.
>
> ...
>
> If you need a TL;DR for the TL;DR, I’ll just [quote](https://snarky.ca/why-i-dont-like-semver/) Python Steering Council Member and packaging expert Brett Cannon:
>
>Libraries/packages should be setting a floor, and if necessary excluding known buggy versions, but otherwise don’t cap the maximum version as you can’t predict future compatibility



@smackesey Hear you loud & clear! I just opened a discussion to summarize some of our thinking on dependencies over the past few months: https://github.com/dbt-labs/dbt-core/discussions/6495

Resolving patch:
diff --git a/core/dbt/semver.py b/core/dbt/semver.py
--- a/core/dbt/semver.py
+++ b/core/dbt/semver.py
@@ -1,10 +1,7 @@
 from dataclasses import dataclass
 import re
-import warnings
 from typing import List
 
-from packaging import version as packaging_version
-
 from dbt.exceptions import VersionsNotCompatibleError
 import dbt.utils
 
@@ -70,6 +67,11 @@ class VersionSpecification(dbtClassMixin):
 _VERSION_REGEX = re.compile(_VERSION_REGEX_PAT_STR, re.VERBOSE)
 
 
+def _cmp(a, b):
+    """Return negative if a<b, zero if a==b, positive if a>b."""
+    return (a > b) - (a < b)
+
+
 @dataclass
 class VersionSpecifier(VersionSpecification):
     def to_version_string(self, skip_matcher=False):
@@ -142,13 +144,19 @@ def compare(self, other):
                     return 1
                 if b is None:
                     return -1
-            # This suppresses the LegacyVersion deprecation warning
-            with warnings.catch_warnings():
-                warnings.simplefilter("ignore", category=DeprecationWarning)
-                if packaging_version.parse(a) > packaging_version.parse(b):
+
+                # Check the prerelease component only
+                prcmp = self._nat_cmp(a, b)
+                if prcmp != 0:  # either -1 or 1
+                    return prcmp
+                # else is equal and will fall through
+
+            else:  # major/minor/patch, should all be numbers
+                if a > b:
                     return 1
-                elif packaging_version.parse(a) < packaging_version.parse(b):
+                elif a < b:
                     return -1
+                # else is equal and will fall through
 
         equal = (
             self.matcher == Matchers.GREATER_THAN_OR_EQUAL
@@ -212,6 +220,29 @@ def is_upper_bound(self):
     def is_exact(self):
         return self.matcher == Matchers.EXACT
 
+    @classmethod
+    def _nat_cmp(cls, a, b):
+        def cmp_prerelease_tag(a, b):
+            if isinstance(a, int) and isinstance(b, int):
+                return _cmp(a, b)
+            elif isinstance(a, int):
+                return -1
+            elif isinstance(b, int):
+                return 1
+            else:
+                return _cmp(a, b)
+
+        a, b = a or "", b or ""
+        a_parts, b_parts = a.split("."), b.split(".")
+        a_parts = [int(x) if re.match(r"^\d+$", x) else x for x in a_parts]
+        b_parts = [int(x) if re.match(r"^\d+$", x) else x for x in b_parts]
+        for sub_a, sub_b in zip(a_parts, b_parts):
+            cmp_result = cmp_prerelease_tag(sub_a, sub_b)
+            if cmp_result != 0:
+                return cmp_result
+        else:
+            return _cmp(len(a), len(b))
+
 
 @dataclass
 class VersionRange:
diff --git a/core/setup.py b/core/setup.py
--- a/core/setup.py
+++ b/core/setup.py
@@ -58,7 +58,7 @@
         "minimal-snowplow-tracker==0.0.2",
         "networkx>=2.3,<2.8.1;python_version<'3.8'",
         "networkx>=2.3,<3;python_version>='3.8'",
-        "packaging>=20.9,<22.0",
+        "packaging>20.9",
         "sqlparse>=0.2.3,<0.5",
         "dbt-extractor~=0.4.1",
         "typing-extensions>=3.7.4",


Test patch:
diff --git a/test/unit/test_semver.py b/test/unit/test_semver.py
--- a/test/unit/test_semver.py
+++ b/test/unit/test_semver.py
@@ -201,12 +201,16 @@ def test__resolve_to_specific_version(self):
             '1.1.0')
 
     def test__filter_installable(self):
-        assert filter_installable(
+        installable = filter_installable(
             ['1.1.0',  '1.2.0a1', '1.0.0','2.1.0-alpha','2.2.0asdf','2.1.0','2.2.0','2.2.0-fishtown-beta','2.2.0-2'],
             install_prerelease=True
-        ) == ['1.0.0', '1.1.0', '1.2.0a1','2.1.0-alpha','2.1.0','2.2.0asdf','2.2.0-fishtown-beta','2.2.0-2','2.2.0']
+        )
+        expected = ['1.0.0', '1.1.0', '1.2.0a1','2.1.0-alpha','2.1.0','2.2.0-2','2.2.0asdf','2.2.0-fishtown-beta','2.2.0']
+        assert installable == expected
 
-        assert filter_installable(
+        installable = filter_installable(
             ['1.1.0',  '1.2.0a1', '1.0.0','2.1.0-alpha','2.2.0asdf','2.1.0','2.2.0','2.2.0-fishtown-beta'],
             install_prerelease=False
-        ) == ['1.0.0', '1.1.0','2.1.0','2.2.0']
+        )
+        expected = ['1.0.0', '1.1.0','2.1.0','2.2.0']
+        assert installable == expected