Repository: dbt-labs/dbt-core

Issue:
[CT-473] [Bug] dbt deps still can fail with `'NoneType' object is not subscriptable`
### Is there an existing issue for this?

- [X] I have searched the existing issues

### Current Behavior

It appears that there is still an exception that the current code is not catching on v1.0.5-rc2 -- this one is new!

```
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1333, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 131, in get_available_versions
    response = package(package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 120, in package
    return response["versions"]
TypeError: 'NoneType' object is not subscriptable
```

so something else is sneaking through.

### Expected Behavior

The root cause of this error is caught, logged, and the download retried.

### Steps To Reproduce

_No response_

### Relevant log output

_No response_

### Environment

```markdown
- dbt: 1.0.5-rc2 (dbt Cloud)
```


### What database are you using dbt with?

_No response_

### Additional Context

_No response_


Discussion:
Thanks for the report @barberscott!

I'm trying to wrap my head around how, if `response` is `None`, it still managed to clear all of these checks, rather than encountering one of these many other exceptions:

https://github.com/dbt-labs/dbt-core/blob/5852f17f0b67723df6aced9af69de94788c79bb7/core/dbt/clients/registry.py#L54-L90

Ideally, that exception would trigger a retry; at the very least, it should be the exception we see in the stacktrace, rather than the surprise at trying to resolve `response["versions"]`.

Accessing the raw logs may give a better sense of the narrative, for exactly which steps succeeded in the leadup to the `package` call at the end of the stacktrace. The only thing I can think is that we may need to switch the order of retry and caching, so that `_get_with_retries` -> `_get_cached` rather than `_get_cached` -> `_get_with_retries`.
I poked through the code too and didn't see how that could happen. At the very least, we should check for `if "versions" in response: ... else: <dump information and throw error>`
I do wonder if we should catch all exceptions in connection_exception_retry. We're only catching RequestException and ReadError. Should we also add a catch of Exception? Not catching an exception there shouldn't cause this error as far as I can tell, but just to make sure there's not something we're missing...
Reopening since it looks like, darn, we didn't catch this!

A recent dbt run failed on v1.0.5 with the following error:

```
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1338, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 56, in run
    final_deps = resolve_packages(packages, self.config)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/resolver.py", line 135, in resolve_packages
    target = final[package].resolved().fetch_metadata(config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 150, in resolved
    available = registry.get_available_versions(self.package)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 131, in get_available_versions
    response = package(package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/registry.py", line 120, in package
    return response["versions"]
TypeError: 'NoneType' object is not subscriptable
```

Resolving patch:
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -615,7 +615,7 @@ def _connection_exception_retry(fn, max_attempts: int, attempt: int = 0):
             fire_event(RecordRetryException(exc=exc))
             fire_event(RetryExternalCall(attempt=attempt, max=max_attempts))
             time.sleep(1)
-            _connection_exception_retry(fn, max_attempts, attempt + 1)
+            return _connection_exception_retry(fn, max_attempts, attempt + 1)
         else:
             raise ConnectionException("External connection exception occurred: " + str(exc))
 


Test patch:
diff --git a/tests/unit/test_connection_retries.py b/tests/unit/test_connection_retries.py
new file mode 100644
--- /dev/null
+++ b/tests/unit/test_connection_retries.py
@@ -0,0 +1,59 @@
+import functools
+import pytest
+from requests.exceptions import RequestException
+from dbt.exceptions import ConnectionException
+from dbt.utils import _connection_exception_retry
+
+
+def no_retry_fn():
+    return "success"
+
+
+class TestNoRetries:
+    def test_no_retry(self):
+        fn_to_retry = functools.partial(no_retry_fn)
+        result = _connection_exception_retry(fn_to_retry, 3)
+
+        expected = "success"
+
+        assert result == expected
+
+
+def no_success_fn():
+    raise RequestException("You'll never pass")
+    return "failure"
+
+
+class TestMaxRetries:
+    def test_no_retry(self):
+        fn_to_retry = functools.partial(no_success_fn)
+
+        with pytest.raises(ConnectionException):
+            _connection_exception_retry(fn_to_retry, 3)
+
+
+def single_retry_fn():
+    global counter
+    if counter == 0:
+        counter += 1
+        raise RequestException("You won't pass this one time")
+    elif counter == 1:
+        counter += 1
+        return "success on 2"
+
+    return "How did we get here?"
+
+
+class TestSingleRetry:
+    def test_no_retry(self):
+        global counter
+        counter = 0
+
+        fn_to_retry = functools.partial(single_retry_fn)
+        result = _connection_exception_retry(fn_to_retry, 3)
+        expected = "success on 2"
+
+        # We need to test the return value here, not just that it did not throw an error.
+        # If the value is not being passed it causes cryptic errors
+        assert result == expected
+        assert counter == 2