Repository: dbt-labs/dbt-core

Issue:
Package resolution is inconsistent for prerelease versions
### Describe the bug
The incorrect package is installed using package/version syntax, but the correct package is installed using git/revision syntax.

### Steps To Reproduce

Set packages.yml to install 0.7.6 version of dbtvault.
Run dbt deps.

The wrong version of dbtvault is installed. Below packages.yml is set to 0.7.6 but b1 variation is installed.

Same behaviour on 0.19.1 and 0.20.0



### Expected behavior
Expected that the package version listed is the one installed.

### Screenshots and log output
```
❯ cat packages.yml
packages:
  - package: Datavault-UK/dbtvault
    version: 0.7.6
❯ dbt deps
Running with dbt=0.19.1
No profile "default" found, continuing with no target
Installing Datavault-UK/dbtvault@0.7.6-b1
  Installed from version 0.7.6-b1

Installing fishtown-analytics/dbt_utils@0.6.4
  Installed from version 0.6.4
```
### System information
**Which database are you using dbt with?**
- [ ] postgres
- [ ] redshift
- [ ] bigquery
- [ ] snowflake
- [x] other (specify: Not applicable.)


**The output of `dbt --version`:**
```
❯ dbt --version
installed version: 0.20.0
   latest version: 0.20.0

Up to date!
```

**The operating system you're using:**
OSX, but also occurs in dbt Cloud.



**The output of `python --version`:**
```
❯ python --version
Python 2.7.16
```
### Additional context
Add any other context about the problem here.



Discussion:
Thanks for the report @lbk-fishtown. You're totally right, `dbt deps` with the package registry does not appropriately handle semantic versions with prerelease suffixes. As we continue to add packages to the Hub site, and support extended prerelease testing cycles, I think it's important for us to support this.

I managed to narrow down the issue, landing on the `compare` method in our semver logic:

https://github.com/dbt-labs/dbt/blob/f460d275baa4df8cf8713f27e198a3577d6ea71c/core/dbt/semver.py#L124-L159

This is not quite right—we need to make sure the final version supports range logic—but I can add a conditional like:
```python
        if not (
            self.prerelease == other.prerelease or
            (self.prerelease is None and other.prerelease is None)
        ):
            return 1
```
Then I can manage:
```yml
packages:
  - package: Datavault-UK/dbtvault
    version: 0.7.6
```
```
$ dbt deps
Running with dbt=0.20.0
Installing Datavault-UK/dbtvault@0.7.6
  Installed from version 0.7.6
Installing dbt-labs/dbt_utils@0.7.0
  Installed from version 0.7.0
```
While still supporting:
```yml
packages:
  - package: Datavault-UK/dbtvault
    version: 0.7.6-b1
```
```
$ dbt deps
Running with dbt=0.20.0-rc1
Installing Datavault-UK/dbtvault@0.7.6-b1
  Installed from version 0.7.6-b1
Installing fishtown-analytics/dbt_utils@0.6.4
  Installed from version 0.6.4
```

Resolving patch:
diff --git a/core/dbt/contracts/project.py b/core/dbt/contracts/project.py
--- a/core/dbt/contracts/project.py
+++ b/core/dbt/contracts/project.py
@@ -83,6 +83,7 @@ def get_revisions(self) -> List[str]:
 class RegistryPackage(Package):
     package: str
     version: Union[RawVersion, List[RawVersion]]
+    install_prerelease: Optional[bool] = False
 
     def get_versions(self) -> List[str]:
         if isinstance(self.version, list):
diff --git a/core/dbt/deps/registry.py b/core/dbt/deps/registry.py
--- a/core/dbt/deps/registry.py
+++ b/core/dbt/deps/registry.py
@@ -71,10 +71,14 @@ class RegistryUnpinnedPackage(
     RegistryPackageMixin, UnpinnedPackage[RegistryPinnedPackage]
 ):
     def __init__(
-        self, package: str, versions: List[semver.VersionSpecifier]
+        self,
+        package: str,
+        versions: List[semver.VersionSpecifier],
+        install_prerelease: bool
     ) -> None:
         super().__init__(package)
         self.versions = versions
+        self.install_prerelease = install_prerelease
 
     def _check_in_index(self):
         index = registry.index_cached()
@@ -91,13 +95,18 @@ def from_contract(
             semver.VersionSpecifier.from_version_string(v)
             for v in raw_version
         ]
-        return cls(package=contract.package, versions=versions)
+        return cls(
+            package=contract.package,
+            versions=versions,
+            install_prerelease=contract.install_prerelease
+        )
 
     def incorporate(
         self, other: 'RegistryUnpinnedPackage'
     ) -> 'RegistryUnpinnedPackage':
         return RegistryUnpinnedPackage(
             package=self.package,
+            install_prerelease=self.install_prerelease,
             versions=self.versions + other.versions,
         )
 
@@ -111,12 +120,16 @@ def resolved(self) -> RegistryPinnedPackage:
             raise DependencyException(new_msg) from e
 
         available = registry.get_available_versions(self.package)
+        installable = semver.filter_installable(
+            available,
+            self.install_prerelease
+        )
 
         # for now, pick a version and then recurse. later on,
         # we'll probably want to traverse multiple options
         # so we can match packages. not going to make a difference
         # right now.
-        target = semver.resolve_to_specific_version(range_, available)
+        target = semver.resolve_to_specific_version(range_, installable)
         if not target:
-            package_version_not_found(self.package, range_, available)
+            package_version_not_found(self.package, range_, installable)
         return RegistryPinnedPackage(package=self.package, version=target)
diff --git a/core/dbt/semver.py b/core/dbt/semver.py
--- a/core/dbt/semver.py
+++ b/core/dbt/semver.py
@@ -1,5 +1,8 @@
 from dataclasses import dataclass
 import re
+from typing import List
+
+from packaging import version as packaging_version
 
 from dbt.exceptions import VersionsNotCompatibleException
 import dbt.utils
@@ -125,12 +128,26 @@ def compare(self, other):
         if self.is_unbounded or other.is_unbounded:
             return 0
 
-        for key in ['major', 'minor', 'patch']:
-            comparison = int(getattr(self, key)) - int(getattr(other, key))
-
-            if comparison > 0:
+        for key in ['major', 'minor', 'patch', 'prerelease']:
+            (a, b) = (getattr(self, key), getattr(other, key))
+            if key == 'prerelease':
+                if a is None and b is None:
+                    continue
+                if a is None:
+                    if self.matcher == Matchers.LESS_THAN:
+                        # If 'a' is not a pre-release but 'b' is, and b must be
+                        # less than a, return -1 to prevent installations of
+                        # pre-releases with greater base version than a
+                        # maximum specified non-pre-release version.
+                        return -1
+                    # Otherwise, stable releases are considered greater than
+                    # pre-release
+                    return 1
+                if b is None:
+                    return -1
+            if packaging_version.parse(a) > packaging_version.parse(b):
                 return 1
-            elif comparison < 0:
+            elif packaging_version.parse(a) < packaging_version.parse(b):
                 return -1
 
         equal = ((self.matcher == Matchers.GREATER_THAN_OR_EQUAL and
@@ -408,10 +425,23 @@ def resolve_to_specific_version(requested_range, available_versions):
         version = VersionSpecifier.from_version_string(version_string)
 
         if(versions_compatible(version,
-                               requested_range.start,
-                               requested_range.end) and
+           requested_range.start, requested_range.end) and
            (max_version is None or max_version.compare(version) < 0)):
             max_version = version
             max_version_string = version_string
 
     return max_version_string
+
+
+def filter_installable(
+        versions: List[str],
+        install_prerelease: bool
+) -> List[str]:
+    if install_prerelease:
+        return versions
+    installable = []
+    for version_string in versions:
+        version = VersionSpecifier.from_version_string(version_string)
+        if not version.prerelease:
+            installable.append(version_string)
+    return installable


Test patch:
diff --git a/test/unit/test_deps.py b/test/unit/test_deps.py
--- a/test/unit/test_deps.py
+++ b/test/unit/test_deps.py
@@ -121,7 +121,7 @@ def setUp(self):
             'fishtown-analytics-test/a',
         ]
         self.get_available_versions.return_value = [
-            '0.1.2', '0.1.3'
+            '0.1.2', '0.1.3', '0.1.4a1'
         ]
         self.package_version.return_value = {
             'id': 'fishtown-analytics-test/a/0.1.2',
@@ -303,6 +303,89 @@ def test_resolve_ranges(self):
         self.assertEqual(c_pinned.version, '0.1.2')
         self.assertEqual(c_pinned.source_type(), 'hub')
 
+    def test_resolve_ranges_install_prerelease_default_false(self):
+        a_contract = RegistryPackage(
+            package='fishtown-analytics-test/a',
+            version='>0.1.2'
+        )
+        b_contract = RegistryPackage(
+            package='fishtown-analytics-test/a',
+            version='<0.1.5'
+        )
+        a = RegistryUnpinnedPackage.from_contract(a_contract)
+        b = RegistryUnpinnedPackage.from_contract(b_contract)
+        c = a.incorporate(b)
+
+        self.assertEqual(c.package, 'fishtown-analytics-test/a')
+        self.assertEqual(
+            c.versions,
+            [
+                VersionSpecifier(
+                    build=None,
+                    major='0',
+                    matcher='>',
+                    minor='1',
+                    patch='2',
+                    prerelease=None,
+                ),
+                VersionSpecifier(
+                    build=None,
+                    major='0',
+                    matcher='<',
+                    minor='1',
+                    patch='5',
+                    prerelease=None,
+                ),
+            ]
+        )
+
+        c_pinned = c.resolved()
+        self.assertEqual(c_pinned.package, 'fishtown-analytics-test/a')
+        self.assertEqual(c_pinned.version, '0.1.3')
+        self.assertEqual(c_pinned.source_type(), 'hub')
+
+    def test_resolve_ranges_install_prerelease_true(self):
+        a_contract = RegistryPackage(
+            package='fishtown-analytics-test/a',
+            version='>0.1.2',
+            install_prerelease=True
+        )
+        b_contract = RegistryPackage(
+            package='fishtown-analytics-test/a',
+            version='<0.1.5'
+        )
+        a = RegistryUnpinnedPackage.from_contract(a_contract)
+        b = RegistryUnpinnedPackage.from_contract(b_contract)
+        c = a.incorporate(b)
+
+        self.assertEqual(c.package, 'fishtown-analytics-test/a')
+        self.assertEqual(
+            c.versions,
+            [
+                VersionSpecifier(
+                    build=None,
+                    major='0',
+                    matcher='>',
+                    minor='1',
+                    patch='2',
+                    prerelease=None,
+                ),
+                VersionSpecifier(
+                    build=None,
+                    major='0',
+                    matcher='<',
+                    minor='1',
+                    patch='5',
+                    prerelease=None,
+                ),
+            ]
+        )
+
+        c_pinned = c.resolved()
+        self.assertEqual(c_pinned.package, 'fishtown-analytics-test/a')
+        self.assertEqual(c_pinned.version, '0.1.4a1')
+        self.assertEqual(c_pinned.source_type(), 'hub')
+
 
 class MockRegistry:
     def __init__(self, packages):
diff --git a/test/unit/test_semver.py b/test/unit/test_semver.py
--- a/test/unit/test_semver.py
+++ b/test/unit/test_semver.py
@@ -4,7 +4,7 @@
 from dbt.exceptions import VersionsNotCompatibleException
 from dbt.semver import VersionSpecifier, UnboundedVersionSpecifier, \
     VersionRange, reduce_versions, versions_compatible, \
-    resolve_to_specific_version
+    resolve_to_specific_version, filter_installable
 
 
 def create_range(start_version_string, end_version_string):
@@ -42,6 +42,8 @@ def test__versions_compatible(self):
             versions_compatible('0.0.1', '0.0.2'))
         self.assertTrue(
             versions_compatible('>0.0.1', '0.0.2'))
+        self.assertFalse(
+            versions_compatible('0.4.5a1', '0.4.5a2'))
 
     def test__reduce_versions(self):
         self.assertVersionSetResult(
@@ -134,3 +136,50 @@ def test__resolve_to_specific_version(self):
                 create_range(None, '<=0.0.5'),
                 ['0.0.3', '0.1.4', '0.0.5']),
             '0.0.5')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('=0.4.5a2', '=0.4.5a2'),
+                ['0.4.5a1', '0.4.5a2']),
+            '0.4.5a2')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('=0.7.6', '=0.7.6'),
+                ['0.7.6-b1', '0.7.6']),
+            '0.7.6')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('>=1.0.0', None),
+                ['1.0.0', '1.1.0a1', '1.1.0', '1.2.0a1']),
+            '1.2.0a1')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('>=1.0.0', '<1.2.0'),
+                ['1.0.0', '1.1.0a1', '1.1.0', '1.2.0a1']),
+            '1.1.0')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('>=1.0.0', None),
+                ['1.0.0', '1.1.0a1', '1.1.0', '1.2.0a1', '1.2.0']),
+            '1.2.0')
+
+        self.assertEqual(
+            resolve_to_specific_version(
+                create_range('>=1.0.0', '<1.2.0'),
+                ['1.0.0', '1.1.0a1', '1.1.0', '1.2.0a1', '1.2.0']),
+            '1.1.0')
+
+    def test__filter_installable(self):
+        assert filter_installable(
+            ['1.0.0', '1.1.0', '1.2.0a1'],
+            install_prerelease=True
+        ) == ['1.0.0', '1.1.0', '1.2.0a1']
+
+        assert filter_installable(
+            ['1.0.0', '1.1.0', '1.2.0a1'],
+            install_prerelease=False
+        ) == ['1.0.0', '1.1.0']