Repository: dbt-labs/dbt-core

Issue:
[Bug] dbt deps sometimes takes >90sec to complete
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

Over the last couple of weeks, I've noticed that `dbt deps` sometimes takes >90 seconds between `Running with dbt==0.21.0` and the first `Installing...` line.

It's sporadic, and when I re-run to try and capture it performance improves, which makes me suspect it's some sort of cold cache thing. 

### Expected Behavior

<img alt="Let's go faster" src="https://media.giphy.com/media/3ohuPuRnWLwIsCuHPa/giphy.gif">


### Steps To Reproduce

With this packages.yml file (from the internal-analyics project): 
```
packages:
  - local: '../dbt-utils'
  - package: dbt-labs/snowplow
    version: 0.13.0
  - package: dbt-labs/codegen
    version: 0.4.0
  - package: tailsdotcom/dbt_artifacts
    version: 0.5.0
  - package: fivetran/hubspot
    version: 0.4.0
  - package: fivetran/zendesk_source
    version: 0.4.0
  - package: dbt-labs/audit_helper
    version: 0.4.0
```

run `dbt deps`. I was on the VPN, but from memory I've experienced it off the VPN as well. 

Note that this combination will ultimately result in a duplicate package error due to codegen depending on utils as well, but I don't think that's relevant. 

### Relevant log output

```shell
https://www.loom.com/share/95a1c1f62c304673a4649d609aa98781
```


### Environment

```markdown
- OS: macOS 11.6.1
- Python: Python 3.8.10
- dbt: I've noticed this on 0.21.0 as well as various prerelease versions of 1.0
```


### What database are you using dbt with?

snowflake

### Additional Context

_No response_


Discussion:
@joellabes Thanks for opening! I've noticed this too.

I believe the significant delay comes from the fact that dbt needs to make _many_ hub API requests in order to "discover" available package versions:
```
2021-11-29 10:25:41.507359 (MainThread): Set downloads directory='/var/folders/7h/hj5_fw9j291c58hwfdvy5xbm0000gp/T/dbt-downloads-nlq7cw6p'
2021-11-29 10:25:41.507605 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/index.json
2021-11-29 10:25:41.786837 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/index.json 200
2021-11-29 10:25:41.787967 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/dbt-labs/dbt_utils.json
2021-11-29 10:25:42.027193 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/dbt-labs/dbt_utils.json 200
2021-11-29 10:25:42.061386 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/dbt-labs/dbt_utils/0.7.1.json
2021-11-29 10:25:42.266005 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/dbt-labs/dbt_utils/0.7.1.json 200
2021-11-29 10:25:42.266890 (MainThread): Making package registry request: GET https://hub.getdbt.com/api/v1/dbt-labs/snowplow.json
2021-11-29 10:25:42.856051 (MainThread): Response from registry: GET https://hub.getdbt.com/api/v1/dbt-labs/snowplow.json 200
... and so on ...
```

(Could you check your debug logs (`logs/dbt.log`), or try running with `dbt --debug deps`, to confirm that's where the delay seems to be?)

My sense is that:
- The hub API can be quite fast or quite slow (perhaps varied based on overall traffic?)
- I don't think this changed at all in v0.21. Anecdotally, I see the same number of hub API requests when I run with v0.20.2 and v0.21.0
- Requests for common dependencies (e.g. `dbt_utils`) are repeated several times. We could probably find ways to reduce the number of requests that dbt-core is making to the Hub API, or—though this is a heavier lift—find ways to make those requests in parallel
@jtcohen6 yes there seems to be 1min 38sec of registry requests (between 2:18:49 and 2:20:27) before any package is installed - [dbt.log](https://github.com/dbt-labs/dbt-core/files/7885717/dbt.log). 2min 5 sec before the `upgrades available` message is displayed at 2:20:54. 

Agreed that this isn't a ~1.0~ recent regression. it had been annoying me for a while but I hadn't gotten around to documenting it. 

Parallel requests would be interesting for the installs towards the end, but – at least for our project with 7 packages – it seems like the real win would be if the hub requests could be a batch request. If we could pass up a request containing all the Hub-hosted packages at once, and get back an array of tarballs + hashes then that'd be nicer. 

I just had a bit of a look at the objects being returned from the [hub API](https://hub.getdbt.com/api/v1/tailsdotcom/dbt_artifacts.json), and it looks like that sort of change would mean all the version and dependency logic would start to live on the server instead of on the client. Do we have an appetite for that, or is it a *feature* that the client does all this locally? (or is this what you meant when you said parallelising would be a heavier lift)
I don't feel so strongly about where version/dependency resolution happens. It's a bit strange that the logic for it lives in `dbt-core`; it might be a little bit stranger if it lived within hub.getdbt.com. We've talked about splitting this out into a separate package (`dbt-deps`?), anticipating a future in which we need its capabilities in multiple places and want to avoid duplicating code.

From a functional standpoint, if all version/dependency resolution were to happen via a single batch call to the Hub API, what would this look like? Send up the full contents of `packages.yml`, and receive back the link to all relevant tarballs? I'm worried that this might not work for package specifications that mix `local` packages, `git` packages, and Hub-hosted `package` entries—thinking, too, that the first two can declare their own dependencies on the third.
> Send up the full contents of `packages.yml`, and receive back the link to all relevant tarballs

I was thinking just the ones identified as hub-hosted, but whether it's discarded client-side or server-side doesn't _really_ matter.



> the first two can declare their own dependencies on the third

But i had forgotten about this wrinkle. Is the same true in reverse (a hub package declaring a git dependency)? If not, then maybe we resolve all of those first, then we know the whole list of hub packages we need to dig up.
Adding this here as @joellabes pointed out there's a related issue. Almost a good 5 minutes to install the [ad_reporting](https://hub.getdbt.com/fivetran/ad_reporting/latest/) super package.

https://www.loom.com/share/b8663b40c1d548ff91fe8cf04042a670


Resolving patch:
diff --git a/core/dbt/clients/registry.py b/core/dbt/clients/registry.py
--- a/core/dbt/clients/registry.py
+++ b/core/dbt/clients/registry.py
@@ -1,7 +1,17 @@
 import functools
+from typing import Any, Dict, List
 import requests
 from dbt.events.functions import fire_event
-from dbt.events.types import RegistryProgressMakingGETRequest, RegistryProgressGETResponse
+from dbt.events.types import (
+    RegistryProgressMakingGETRequest,
+    RegistryProgressGETResponse,
+    RegistryIndexProgressMakingGETRequest,
+    RegistryIndexProgressGETResponse,
+    RegistryResponseUnexpectedType,
+    RegistryResponseMissingTopKeys,
+    RegistryResponseMissingNestedKeys,
+    RegistryResponseExtraNestedKeys,
+)
 from dbt.utils import memoized, _connection_exception_retry as connection_exception_retry
 from dbt import deprecations
 import os
@@ -12,55 +22,77 @@
     DEFAULT_REGISTRY_BASE_URL = "https://hub.getdbt.com/"
 
 
-def _get_url(url, registry_base_url=None):
+def _get_url(name, registry_base_url=None):
     if registry_base_url is None:
         registry_base_url = DEFAULT_REGISTRY_BASE_URL
+    url = "api/v1/{}.json".format(name)
 
     return "{}{}".format(registry_base_url, url)
 
 
-def _get_with_retries(path, registry_base_url=None):
-    get_fn = functools.partial(_get, path, registry_base_url)
+def _get_with_retries(package_name, registry_base_url=None):
+    get_fn = functools.partial(_get_cached, package_name, registry_base_url)
     return connection_exception_retry(get_fn, 5)
 
 
-def _get(path, registry_base_url=None):
-    url = _get_url(path, registry_base_url)
+def _get(package_name, registry_base_url=None):
+    url = _get_url(package_name, registry_base_url)
     fire_event(RegistryProgressMakingGETRequest(url=url))
+    # all exceptions from requests get caught in the retry logic so no need to wrap this here
     resp = requests.get(url, timeout=30)
     fire_event(RegistryProgressGETResponse(url=url, resp_code=resp.status_code))
     resp.raise_for_status()
 
-    # It is unexpected for the content of the response to be None so if it is, raising this error
-    # will cause this function to retry (if called within _get_with_retries) and hopefully get
-    # a response.  This seems to happen when there's an issue with the Hub.
+    # The response should always be a dictionary.  Anything else is unexpected, raise error.
+    # Raising this error will cause this function to retry (if called within _get_with_retries)
+    # and hopefully get a valid response.  This seems to happen when there's an issue with the Hub.
+    # Since we control what we expect the HUB to return, this is safe.
     # See https://github.com/dbt-labs/dbt-core/issues/4577
-    if resp.json() is None:
-        raise requests.exceptions.ContentDecodingError(
-            "Request error: The response is None", response=resp
-        )
-    return resp.json()
-
-
-def index(registry_base_url=None):
-    return _get_with_retries("api/v1/index.json", registry_base_url)
-
-
-index_cached = memoized(index)
-
-
-def packages(registry_base_url=None):
-    return _get_with_retries("api/v1/packages.json", registry_base_url)
-
+    # and https://github.com/dbt-labs/dbt-core/issues/4849
+    response = resp.json()
 
-def package(name, registry_base_url=None):
-    response = _get_with_retries("api/v1/{}.json".format(name), registry_base_url)
+    if not isinstance(response, dict):  # This will also catch Nonetype
+        error_msg = (
+            f"Request error: Expected a response type of <dict> but got {type(response)} instead"
+        )
+        fire_event(RegistryResponseUnexpectedType(response=response))
+        raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
+
+    # check for expected top level keys
+    expected_keys = {"name", "versions"}
+    if not expected_keys.issubset(response):
+        error_msg = (
+            f"Request error: Expected the response to contain keys {expected_keys} "
+            f"but is missing {expected_keys.difference(set(response))}"
+        )
+        fire_event(RegistryResponseMissingTopKeys(response=response))
+        raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
+
+    # check for the keys we need nested under each version
+    expected_version_keys = {"name", "packages", "downloads"}
+    all_keys = set().union(*(response["versions"][d] for d in response["versions"]))
+    if not expected_version_keys.issubset(all_keys):
+        error_msg = (
+            "Request error: Expected the response for the version to contain keys "
+            f"{expected_version_keys} but is missing {expected_version_keys.difference(all_keys)}"
+        )
+        fire_event(RegistryResponseMissingNestedKeys(response=response))
+        raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
+
+    # all version responses should contain identical keys.
+    has_extra_keys = set().difference(*(response["versions"][d] for d in response["versions"]))
+    if has_extra_keys:
+        error_msg = (
+            "Request error: Keys for all versions do not match.  Found extra key(s) "
+            f"of {has_extra_keys}."
+        )
+        fire_event(RegistryResponseExtraNestedKeys(response=response))
+        raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
 
     # Either redirectnamespace or redirectname in the JSON response indicate a redirect
     # redirectnamespace redirects based on package ownership
     # redirectname redirects based on package name
     # Both can be present at the same time, or neither. Fails gracefully to old name
-
     if ("redirectnamespace" in response) or ("redirectname" in response):
 
         if ("redirectnamespace" in response) and response["redirectnamespace"] is not None:
@@ -74,15 +106,59 @@ def package(name, registry_base_url=None):
             use_name = response["name"]
 
         new_nwo = use_namespace + "/" + use_name
-        deprecations.warn("package-redirect", old_name=name, new_name=new_nwo)
+        deprecations.warn("package-redirect", old_name=package_name, new_name=new_nwo)
+
+    return response
+
+
+_get_cached = memoized(_get)
+
+
+def package(package_name, registry_base_url=None) -> Dict[str, Any]:
+    # returns a dictionary of metadata for all versions of a package
+    response = _get_with_retries(package_name, registry_base_url)
+    return response["versions"]
+
+
+def package_version(package_name, version, registry_base_url=None) -> Dict[str, Any]:
+    # returns the metadata of a specific version of a package
+    response = package(package_name, registry_base_url)
+    return response[version]
+
+
+def get_available_versions(package_name) -> List["str"]:
+    # returns a list of all available versions of a package
+    response = package(package_name)
+    return list(response)
+
+
+def _get_index(registry_base_url=None):
+
+    url = _get_url("index", registry_base_url)
+    fire_event(RegistryIndexProgressMakingGETRequest(url=url))
+    # all exceptions from requests get caught in the retry logic so no need to wrap this here
+    resp = requests.get(url, timeout=30)
+    fire_event(RegistryIndexProgressGETResponse(url=url, resp_code=resp.status_code))
+    resp.raise_for_status()
+
+    # The response should be a list.  Anything else is unexpected, raise an error.
+    # Raising this error will cause this function to retry and hopefully get a valid response.
+
+    response = resp.json()
+
+    if not isinstance(response, list):  # This will also catch Nonetype
+        error_msg = (
+            f"Request error: The response type of {type(response)} is not valid: {resp.text}"
+        )
+        raise requests.exceptions.ContentDecodingError(error_msg, response=resp)
 
     return response
 
 
-def package_version(name, version, registry_base_url=None):
-    return _get_with_retries("api/v1/{}/{}.json".format(name, version), registry_base_url)
+def index(registry_base_url=None) -> List[str]:
+    # this returns a list of all packages on the Hub
+    get_index_fn = functools.partial(_get_index, registry_base_url)
+    return connection_exception_retry(get_index_fn, 5)
 
 
-def get_available_versions(name):
-    response = package(name)
-    return list(response["versions"])
+index_cached = memoized(index)
diff --git a/core/dbt/events/types.py b/core/dbt/events/types.py
--- a/core/dbt/events/types.py
+++ b/core/dbt/events/types.py
@@ -291,6 +291,25 @@ def message(self) -> str:
         return f"  Checked out at {self.end_sha}."
 
 
+@dataclass
+class RegistryIndexProgressMakingGETRequest(DebugLevel):
+    url: str
+    code: str = "M022"
+
+    def message(self) -> str:
+        return f"Making package index registry request: GET {self.url}"
+
+
+@dataclass
+class RegistryIndexProgressGETResponse(DebugLevel):
+    url: str
+    resp_code: int
+    code: str = "M023"
+
+    def message(self) -> str:
+        return f"Response from registry index: GET {self.url} {self.resp_code}"
+
+
 @dataclass
 class RegistryProgressMakingGETRequest(DebugLevel):
     url: str
@@ -310,6 +329,45 @@ def message(self) -> str:
         return f"Response from registry: GET {self.url} {self.resp_code}"
 
 
+@dataclass
+class RegistryResponseUnexpectedType(DebugLevel):
+    response: str
+    code: str = "M024"
+
+    def message(self) -> str:
+        return f"Response was None: {self.response}"
+
+
+@dataclass
+class RegistryResponseMissingTopKeys(DebugLevel):
+    response: str
+    code: str = "M025"
+
+    def message(self) -> str:
+        # expected/actual keys logged in exception
+        return f"Response missing top level keys: {self.response}"
+
+
+@dataclass
+class RegistryResponseMissingNestedKeys(DebugLevel):
+    response: str
+    code: str = "M026"
+
+    def message(self) -> str:
+        # expected/actual keys logged in exception
+        return f"Response missing nested keys: {self.response}"
+
+
+@dataclass
+class RegistryResponseExtraNestedKeys(DebugLevel):
+    response: str
+    code: str = "M027"
+
+    def message(self) -> str:
+        # expected/actual keys logged in exception
+        return f"Response contained inconsistent keys: {self.response}"
+
+
 # TODO this was actually `logger.exception(...)` not `logger.error(...)`
 @dataclass
 class SystemErrorRetrievingModTime(ErrorLevel):
@@ -2422,6 +2480,14 @@ def message(self) -> str:
     GitNothingToDo(sha="")
     GitProgressUpdatedCheckoutRange(start_sha="", end_sha="")
     GitProgressCheckedOutAt(end_sha="")
+    RegistryIndexProgressMakingGETRequest(url="")
+    RegistryIndexProgressGETResponse(url="", resp_code=1234)
+    RegistryProgressMakingGETRequest(url="")
+    RegistryProgressGETResponse(url="", resp_code=1234)
+    RegistryResponseUnexpectedType(response=""),
+    RegistryResponseMissingTopKeys(response=""),
+    RegistryResponseMissingNestedKeys(response=""),
+    RegistryResponseExtraNestedKeys(response=""),
     SystemErrorRetrievingModTime(path="")
     SystemCouldNotWrite(path="", reason="", exc=Exception(""))
     SystemExecutingCmd(cmd=[""])


Test patch:
diff --git a/test/unit/test_events.py b/test/unit/test_events.py
--- a/test/unit/test_events.py
+++ b/test/unit/test_events.py
@@ -402,6 +402,12 @@ def MockNode():
     PrintDebugStackTrace(),
     MainReportArgs(args={}),
     RegistryProgressMakingGETRequest(url=''),
+    RegistryIndexProgressMakingGETRequest(url=""),
+    RegistryIndexProgressGETResponse(url="", resp_code=1),
+    RegistryResponseUnexpectedType(response=""),
+    RegistryResponseMissingTopKeys(response=""),
+    RegistryResponseMissingNestedKeys(response=""),
+    RegistryResponseExtraNestedKeys(response=""),
     DepsUTD(),
     PartialParsingNotEnabled(),
     SQlRunnerException(exc=Exception('')),
diff --git a/tests/functional/permission/test_permissions.py b/tests/functional/permission/test_permissions.py
--- a/tests/functional/permission/test_permissions.py
+++ b/tests/functional/permission/test_permissions.py
@@ -64,7 +64,6 @@ def test_create_schema_permissions(
         project,
     ):
         # now it should work!
-        # breakpoint()
         project.run_sql("grant create on database {} to noaccess".format(project.database))
         project.run_sql(
             'grant usage, create on schema "{}" to noaccess'.format(project.test_schema)