Repository: dbt-labs/dbt-core

Issue:
[CT-1849] [Bug] Unhandled exception in dbt deps with a bad/truncated download
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

During a github outage, `dbt deps` failed presumably as a result of codeload.github.com sending us a truncated download of a package gzip.  We appear not to handle a failure in the gzip decompress and deps fails overall instead of retrying the download.

### Expected Behavior

Deps would retry the download per its normal retry behavior and not crash/throw unexpectedly.

### Steps To Reproduce

Intermittent -- not reproducible on demand.

### Relevant log output

```shell
Traceback (most recent call last):
  File "/usr/src/app/sinter/clients/dbt.py", line 1417, in call
    dbt_main.handle(command + extra_args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 152, in handle
    res, success = handle_and_check(args)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 192, in handle_and_check
    task, res = run_from_args(parsed)
  File "/usr/local/lib/python3.8/dist-packages/dbt/main.py", line 246, in run_from_args
    results = task.run()
  File "/usr/local/lib/python3.8/dist-packages/dbt/task/deps.py", line 67, in run
    package.install(self.config, renderer)
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 83, in install
    connection_exception_retry(download_untar_fn, 5)
  File "/usr/local/lib/python3.8/dist-packages/dbt/utils.py", line 608, in _connection_exception_retry
    return fn()
  File "/usr/local/lib/python3.8/dist-packages/dbt/deps/registry.py", line 95, in download_and_untar
    system.untar_package(tar_path, deps_path, package_name)
  File "/usr/local/lib/python3.8/dist-packages/dbt/clients/system.py", line 489, in untar_package
    tarball.extractall(dest_dir)
  File "/usr/lib/python3.8/tarfile.py", line 2028, in extractall
    self.extract(tarinfo, path, set_attrs=not tarinfo.isdir(),
  File "/usr/lib/python3.8/tarfile.py", line 2069, in extract
    self._extract_member(tarinfo, os.path.join(path, tarinfo.name),
  File "/usr/lib/python3.8/tarfile.py", line 2141, in _extract_member
    self.makefile(tarinfo, targetpath)
  File "/usr/lib/python3.8/tarfile.py", line 2190, in makefile
    copyfileobj(source, target, tarinfo.size, ReadError, bufsize)
  File "/usr/lib/python3.8/tarfile.py", line 247, in copyfileobj
    buf = src.read(bufsize)
  File "/usr/lib/python3.8/gzip.py", line 292, in read
    return self._buffer.read(size)
  File "/usr/lib/python3.8/_compression.py", line 68, in readinto
    data = self.read(len(byte_view))
  File "/usr/lib/python3.8/gzip.py", line 498, in read
    raise EOFError("Compressed file ended before the "
EOFError: Compressed file ended before the end-of-stream marker was reached
```
```


### Environment

_No response_

### Which database adapter are you using with dbt?

other (mention it in "Additional Context")

### Additional Context

Observed in dbt Cloud.


Discussion:
Thanks @barberscott! We already wrap this in retry logic:
https://github.com/dbt-labs/dbt-core/blob/99f27de9340002f43337eab08625a76869ed074d/core/dbt/deps/base.py#L114-L129

I think the relevant detail here is that `_connection_exception_retry` only retries on `RequestException` or `ReadError`:

https://github.com/dbt-labs/dbt-core/blob/99f27de9340002f43337eab08625a76869ed074d/core/dbt/utils.py#L608-L620

And this is an `EOFError`! Simplest thing here is to add it to the `except` list above.

A more thorough approach would see if there are any other exceptions that might crop up along the way there (from `tarfile.extractall` through `gzip.read`).
We haven't seen other exception types in the wild. Let's apply the narrower fix for now: adding `EOFError` to the set of retryable exception types in `_connection_exception_retry`

Resolving patch:
diff --git a/core/dbt/utils.py b/core/dbt/utils.py
--- a/core/dbt/utils.py
+++ b/core/dbt/utils.py
@@ -601,6 +601,7 @@ def _connection_exception_retry(fn, max_attempts: int, attempt: int = 0):
     except (
         requests.exceptions.RequestException,
         ReadError,
+        EOFError,
     ) as exc:
         if attempt <= max_attempts - 1:
             dbt.events.functions.fire_event(RecordRetryException(exc=str(exc)))


Test patch:
diff --git a/tests/unit/test_core_dbt_utils.py b/tests/unit/test_core_dbt_utils.py
--- a/tests/unit/test_core_dbt_utils.py
+++ b/tests/unit/test_core_dbt_utils.py
@@ -28,6 +28,11 @@ def test_connection_exception_retry_success_failed_untar(self):
         connection_exception_retry(lambda: Counter._add_with_untar_exception(), 5)
         self.assertEqual(2, counter)  # 2 = original attempt returned ReadError, plus 1 retry
 
+    def test_connection_exception_retry_success_failed_eofexception(self):
+        Counter._reset()
+        connection_exception_retry(lambda: Counter._add_with_eof_exception(), 5)
+        self.assertEqual(2, counter)  # 2 = original attempt returned EOFError, plus 1 retry
+
 
 counter: int = 0
 
@@ -57,6 +62,12 @@ def _add_with_untar_exception():
         if counter < 2:
             raise tarfile.ReadError
 
+    def _add_with_eof_exception():
+        global counter
+        counter += 1
+        if counter < 2:
+            raise EOFError
+
     def _reset():
         global counter
         counter = 0