Repository: dbt-labs/dbt-core

Issue:
[CT-473] [Bug] dbt deps still can fail with `'NoneType' object is not subscriptable`
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

It appears that there is still an exception that the current code is not catching on v1.0.5-rc2 -- this one is new!

```
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1333, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 131, in get_available_versions
    response = package(package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 120, in package
    return response["versions"]
TypeError: 'NoneType' object is not subscriptable
```

so something else is sneaking through.

### Expected Behavior

The root cause of this error is caught, logged, and the download retried.

### Steps To Reproduce

_No response_

### Relevant log output

_No response_

### Environment

```markdown
- dbt: 1.0.5-rc2 (dbt Cloud)
```


### What database are you using dbt with?

_No response_

### Additional Context

_No response_


Discussion:
Thanks for the report @barberscott!

I'm trying to wrap my head around how, if `response` is `None`, it still managed to clear all of these checks, rather than encountering one of these many other exceptions:

https://github.com/dbt-labs/dbt-core/blob/5852f17f0b67723df6aced9af69de94788c79bb7/core/dbt/clients/registry.py#L54-L90

Ideally, that exception would trigger a retry; at the very least, it should be the exception we see in the stacktrace, rather than the surprise at trying to resolve `response["versions"]`.

Accessing the raw logs may give a better sense of the narrative, for exactly which steps succeeded in the leadup to the `package` call at the end of the stacktrace. The only thing I can think is that we may need to switch the order of retry and caching, so that `_get_with_retries` -> `_get_cached` rather than `_get_cached` -> `_get_with_retries`.
I poked through the code too and didn't see how that could happen. At the very least, we should check for `if "versions" in response: ... else: <dump information and throw error>`
I do wonder if we should catch all exceptions in connection_exception_retry. We're only catching RequestException and ReadError. Should we also add a catch of Exception? Not catching an exception there shouldn't cause this error as far as I can tell, but just to make sure there's not something we're missing...

Resolving patch:
diff --git a/core/dbt/clients/registry.py b/core/dbt/clients/registry.py
--- a/core/dbt/clients/registry.py
+++ b/core/dbt/clients/registry.py
@@ -31,7 +31,7 @@ def _get_url(name, registry_base_url=None):
 
 
 def _get_with_retries(package_name, registry_base_url=None):
-    get_fn = functools.partial(_get_cached, package_name, registry_base_url)
+    get_fn = functools.partial(_get, package_name, registry_base_url)
     return connection_exception_retry(get_fn, 5)
 
 
@@ -111,12 +111,12 @@ def _get(package_name, registry_base_url=None):
     return response
 
 
-_get_cached = memoized(_get)
+_get_cached = memoized(_get_with_retries)
 
 
 def package(package_name, registry_base_url=None) -> Dict[str, Any]:
     # returns a dictionary of metadata for all versions of a package
-    response = _get_with_retries(package_name, registry_base_url)
+    response = _get_cached(package_name, registry_base_url)
     return response["versions"]
 
 


Test patch: