Repository: dbt-labs/dbt-core

Issue:
[CT-3107] [Bug] nested dependencies not installed when package is a tarball
### Is this a new bug in dbt-core?

- [X] I believe this is a new bug in dbt-core
- [X] I have searched the existing issues, and I could not find an existing issue for this bug

### Current Behavior

when running `dbt deps` to install a package specified as a tarball, dbt doesn't install nested dependencies (i.e. packages specified in the imported package's `packages.yml` file) as it does when installing a package from local, git or the dbt hub.

### Expected Behavior

consistent behaviour across import methods regarding nested dependencies. dbt should install any dependencies specified in the tarball project's packages.yml file.


### Steps To Reproduce

this can be reproduced by importing the tarball of a package with nested dependencies. In this case, importing dbt_expectations should cause dbt_date to be installed, as its included in the package's dependencies here: https://github.com/calogica/dbt-expectations/blob/0.9.0/packages.yml

Steps:
1. create a `packages.yml` file in a project with the following structure:
``` yaml
packages:
  - tarball: "https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz"
    name: "dbt_expectations"
```
2. run `dbt deps`


running dbt deps will only install dbt_expectations:
```
20:08:55  Running with dbt=1.5.6
20:08:55  Installing dbt_expectations
20:08:56  Installed from tarball (url: https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz)
```
compare this to installing the same package from dbt hub, with the following `packages.yml`:
``` yaml
packages:
  - package: calogica/dbt_expectations
    version: "0.9.0"
```
```
20:14:24  Running with dbt=1.5.6
20:14:24  Installing calogica/dbt_expectations
20:14:25  Installed from version 0.9.0
20:14:25  Up to date!
20:14:25  Installing calogica/dbt_date
20:14:25  Installed from version 0.8.1
20:14:25  Updated version available: 0.9.1
20:14:25  
20:14:25  Updates available for packages: ['calogica/dbt_date']                 
Update your versions in packages.yml, then run dbt deps
```

### Relevant log output

_No response_

### Environment

```markdown
- OS: Mac OS 13.5.2 (22G91)
- Python: 3.9
- dbt: 1.5.6
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
Thanks for reporting this @adamlopez !

I was able to reproduce what you reported for the Tarball URL method and also confirmed that nested dependencies work for all other package installation methods.

See full reproducible example below. Note that all of them except for the Tarball URL method creates two directories:
```
dbt_date                dbt_expectations
```

<details>
<summary>

### Reprex
</summary>

```shell
mkdir -p reprex
cd reprex
cat << EOF > dbt_project.yml
name: "my_project"
version: "1.0.0"
config-version: 2
profile: "sandcastle"

clean-targets:
  - target
  - dbt_packages
  - logs
EOF
```

```shell
cat << EOF > packages.yml
packages:
  # Hub package
  - package: calogica/dbt_expectations
    version: 0.9.0
EOF
dbt clean
dbt deps
echo ""
echo "After Hub package install:"
echo ""
ls dbt_packages
```

```shell
cat << EOF > packages.yml
packages:
  # Git package
  - git: "https://github.com/calogica/dbt-expectations.git"  # git HTTP URL
    revision: 0.9.0  # tag, branch name, or 40-character hash
EOF
dbt clean
dbt deps
echo ""
echo "After Git package install:"
echo ""
ls dbt_packages
```

```shell
cat << EOF > packages.yml
packages:
  # 'Private' package
  - git: "git@github.com:calogica/dbt-expectations.git"  # git SSH URL
    revision: 0.9.0  # tag, branch name, or 40-character hash
EOF
dbt clean
dbt deps
echo ""
echo "After 'Private' package install:"
echo ""
ls dbt_packages
```

```shell
git clone https://github.com/calogica/dbt-expectations.git
cat << EOF > packages.yml
packages:
  # Local package
  - local: dbt-expectations
EOF
dbt clean
dbt deps
echo ""
echo "After Local package install:"
echo ""
ls dbt_packages
```

```shell
cat << EOF > packages.yml
packages:
  # Tarball URL
  - tarball: https://codeload.github.com/calogica/dbt-expectations/tar.gz/0.9.0
    name: 'dbt_expectations'
EOF
dbt clean
dbt deps
echo ""
echo "After Tarball URL install:"
echo ""
ls dbt_packages
```

</details>
Thanks for confirming the issue @dbeatty10 - what's the usual process for fixing these types of issues? Should I just submit a PR with a fix? (i've looked through the contribution guidelines and submitted a CLA)
That would be awesome if you'd be willing to open a PR with a fix @adamlopez 🤩 

Just put `resolves #8621` as the first line of the PR description, and it will automatically link your PR with this issue. All the other relevant details are in the ["Submitting a Pull Request"](https://github.com/dbt-labs/dbt-core/blob/8aaed0e29f9560bc53d9d3e88325a9597318e375/CONTRIBUTING.md#submitting-a-pull-request) of the contributing guide.
@adamlopez I just added the `help_wanted` label which signals to the engineering team within dbt Labs that we're reserving this to be solved by a community contributor (you!) 😎 

Resolving patch:
diff --git a/core/dbt/deps/tarball.py b/core/dbt/deps/tarball.py
--- a/core/dbt/deps/tarball.py
+++ b/core/dbt/deps/tarball.py
@@ -1,7 +1,14 @@
+import functools
+import os
+from pathlib import Path
 from typing import Dict
 
-from dbt.contracts.project import RegistryPackageMetadata, TarballPackage
-from dbt.deps.base import PinnedPackage, UnpinnedPackage
+from dbt.clients import system
+from dbt.config.project import PartialProject
+from dbt.contracts.project import TarballPackage
+from dbt.deps.base import PinnedPackage, UnpinnedPackage, get_downloads_path
+from dbt.exceptions import DependencyError
+from dbt.utils import _connection_exception_retry as connection_exception_retry
 
 
 class TarballPackageMixin:
@@ -20,9 +27,10 @@ def source_type(self) -> str:
 class TarballPinnedPackage(TarballPackageMixin, PinnedPackage):
     def __init__(self, tarball: str, package: str) -> None:
         super().__init__(tarball)
-        # setup to recycle RegistryPinnedPackage fns
         self.package = package
         self.version = "tarball"
+        self.tar_path = os.path.join(Path(get_downloads_path()), self.package)
+        self.untarred_path = f"{self.tar_path}_untarred"
 
     @property
     def name(self):
@@ -31,8 +39,7 @@ def name(self):
     def to_dict(self) -> Dict[str, str]:
         return {
             "tarball": self.tarball,
-            "version": self.version,
-            "package": self.package,
+            "name": self.package,
         }
 
     def get_version(self):
@@ -42,23 +49,38 @@ def nice_version_name(self):
         return f"tarball (url: {self.tarball})"
 
     def _fetch_metadata(self, project, renderer):
-        """
-        recycle RegistryPackageMetadata so that we can use the install and
-        download_and_untar from RegistryPinnedPackage next.
-        build RegistryPackageMetadata from info passed via packages.yml since no
-        'metadata' service exists in this case.
-        """
-
-        dct = {
-            "name": self.package,
-            "packages": [],  # note: required by RegistryPackageMetadata
-            "downloads": {"tarball": self.tarball},
-        }
-
-        return RegistryPackageMetadata.from_dict(dct)
+        """Download and untar the project and parse metadata from the project folder."""
+        download_untar_fn = functools.partial(
+            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name
+        )
+        connection_exception_retry(download_untar_fn, 5)
+
+        tar_contents = os.listdir(self.untarred_path)
+        if len(tar_contents) != 1:
+            raise DependencyError(
+                f"Incorrect structure for package extracted from {self.tarball}."
+                f"The extracted package needs to follow the structure {self.name}/<package_contents>."
+            )
+        child_folder = os.listdir(self.untarred_path)[0]
+
+        self.untarred_path = os.path.join(self.untarred_path, child_folder)
+        partial = PartialProject.from_project_root(self.untarred_path)
+        metadata = partial.render_package_metadata(renderer)
+        metadata.name = self.package if self.package else metadata.name
+        return metadata
 
     def install(self, project, renderer):
-        self._install(project, renderer)
+        download_untar_fn = functools.partial(
+            self.download_and_untar, self.tarball, self.tar_path, self.untarred_path, self.name
+        )
+        connection_exception_retry(download_untar_fn, 5)
+        dest_path = self.get_installation_path(project, renderer)
+        if os.path.exists(dest_path):
+            if system.path_is_symlink(dest_path):
+                system.remove_file(dest_path)
+            else:
+                system.rmdir(dest_path)
+        system.move(self.untarred_path, dest_path)
 
 
 class TarballUnpinnedPackage(TarballPackageMixin, UnpinnedPackage[TarballPinnedPackage]):


Test patch:
diff --git a/tests/functional/dependencies/test_simple_dependency.py b/tests/functional/dependencies/test_simple_dependency.py
--- a/tests/functional/dependencies/test_simple_dependency.py
+++ b/tests/functional/dependencies/test_simple_dependency.py
@@ -276,6 +276,25 @@ def test_simple_dependency_deps(self, project):
         assert len(os.listdir("dbt_packages")) == 2
 
 
+class TestTarballNestedDependencies(object):
+    # this version of dbt_expectations has a dependency on dbt_date, which the
+    # package config handling should detect
+    @pytest.fixture(scope="class")
+    def packages(self):
+        return {
+            "packages": [
+                {
+                    "tarball": "https://github.com/calogica/dbt-expectations/archive/refs/tags/0.9.0.tar.gz",
+                    "name": "dbt_expectations",
+                },
+            ]
+        }
+
+    def test_simple_dependency_deps(self, project):
+        run_dbt(["deps"])
+        assert set(os.listdir("dbt_packages")) == set(["dbt_expectations", "dbt_date"])
+
+
 class DependencyBranchBase(object):
     @pytest.fixture(scope="class", autouse=True)
     def setUp(self, project):
diff --git a/tests/unit/test_deps.py b/tests/unit/test_deps.py
--- a/tests/unit/test_deps.py
+++ b/tests/unit/test_deps.py
@@ -10,6 +10,7 @@
 from dbt.deps.tarball import TarballUnpinnedPackage
 from dbt.deps.registry import RegistryUnpinnedPackage
 from dbt.clients.registry import is_compatible_version
+from dbt.config.renderer import DbtProjectYamlRenderer
 from dbt.deps.resolver import resolve_packages
 from dbt.contracts.project import (
     LocalPackage,
@@ -17,6 +18,7 @@
     GitPackage,
     RegistryPackage,
 )
+from dbt.config.project import PartialProject
 from dbt.contracts.project import PackageConfig
 from dbt.semver import VersionSpecifier
 from dbt.version import get_installed_version
@@ -36,40 +38,109 @@ def test_init(self):
 
 
 class TestTarballPackage(unittest.TestCase):
-    def test_TarballPackage(self):
-        from dbt.contracts.project import RegistryPackageMetadata
-        from mashumaro.exceptions import MissingField
+    class MockMetadata:
+        name = "mock_metadata_name"
+
+    @mock.patch("dbt.config.project.PartialProject.from_project_root")
+    @mock.patch("os.listdir")
+    @mock.patch("dbt.deps.tarball.get_downloads_path")
+    @mock.patch("dbt.clients.system.untar_package")
+    @mock.patch("dbt.clients.system.download")
+    def test_fetch_metadata(
+        self,
+        mock_download,
+        mock_untar_package,
+        mock_get_downloads_path,
+        mock_listdir,
+        mock_from_project_root,
+    ):
+        mock_listdir.return_value = ["one_directory/"]
+        mock_get_downloads_path.return_value = "downloads_path"
+        mock_from_project_root.return_value = object()
+        mock_from_project_root.return_value
+        dict_well_formed_contract = {
+            "tarball": "http://example.com/invalid_url@/package.tar.gz",
+            "name": "my_package",
+        }
+
+        a_contract = TarballPackage.from_dict(dict_well_formed_contract)
+        a = TarballUnpinnedPackage.from_contract(a_contract)
 
-        dict_well_formed_contract = {"tarball": "http://example.com", "name": "my_cool_package"}
+        a_pinned = a.resolved()
+        with mock.patch.object(PartialProject, "from_project_root", return_value=PartialProject):
+            with mock.patch.object(
+                PartialProject, "render_package_metadata", return_value=self.MockMetadata
+            ):
+                metadata = a_pinned.fetch_metadata("", DbtProjectYamlRenderer())
+
+        assert metadata == self.MockMetadata
+        mock_download.assert_called_once_with(
+            "http://example.com/invalid_url@/package.tar.gz", "downloads_path/my_package"
+        )
+        mock_untar_package.assert_called_once_with(
+            "downloads_path/my_package", "downloads_path/my_package_untarred", "my_package"
+        )
+
+    @mock.patch("dbt.config.project.PartialProject.from_project_root")
+    @mock.patch("os.listdir")
+    @mock.patch("dbt.deps.tarball.get_downloads_path")
+    @mock.patch("dbt.clients.system.untar_package")
+    @mock.patch("dbt.clients.system.download")
+    def test_fetch_metadata_fails_on_incorrect_tar_folder_structure(
+        self,
+        mock_download,
+        mock_untar_package,
+        mock_get_downloads_path,
+        mock_listdir,
+        mock_from_project_root,
+    ):
+        mock_listdir.return_value = ["one_directory/", "another_directory/"]
+
+        mock_get_downloads_path.return_value = "downloads_path"
+        mock_from_project_root.return_value = object()
+        mock_from_project_root.return_value
+        dict_well_formed_contract = {
+            "tarball": "http://example.com/invalid_url@/package.tar.gz",
+            "name": "my_package",
+        }
 
+        a_contract = TarballPackage.from_dict(dict_well_formed_contract)
+        a = TarballUnpinnedPackage.from_contract(a_contract)
+
+        a_pinned = a.resolved()
+        with mock.patch.object(PartialProject, "from_project_root", return_value=PartialProject):
+            with mock.patch.object(
+                PartialProject, "render_package_metadata", return_value=self.MockMetadata
+            ):
+                with self.assertRaises(dbt.exceptions.DependencyError):
+                    a_pinned.fetch_metadata("", DbtProjectYamlRenderer())
+
+    @mock.patch("dbt.deps.tarball.get_downloads_path")
+    def test_tarball_package_contract(self, mock_get_downloads_path):
+        dict_well_formed_contract = {
+            "tarball": "http://example.com/invalid_url@/package.tar.gz",
+            "name": "my_cool_package",
+        }
         a_contract = TarballPackage.from_dict(dict_well_formed_contract)
 
         # check contract and resolver
-        self.assertEqual(a_contract.tarball, "http://example.com")
+        self.assertEqual(a_contract.tarball, "http://example.com/invalid_url@/package.tar.gz")
         self.assertEqual(a_contract.name, "my_cool_package")
 
         a = TarballUnpinnedPackage.from_contract(a_contract)
-        self.assertEqual(a.tarball, "http://example.com")
+        self.assertEqual(a.tarball, "http://example.com/invalid_url@/package.tar.gz")
         self.assertEqual(a.package, "my_cool_package")
 
         a_pinned = a.resolved()
         self.assertEqual(a_pinned.source_type(), "tarball")
 
-        # check bad contract (no name) fails
-        dict_missing_name_should_fail_on_contract = {"tarball": "http://example.com"}
+    def test_tarball_package_contract_fails_on_no_name(self):
+        from mashumaro.exceptions import MissingField
 
+        # check bad contract (no name) fails
+        a_contract = {"tarball": "http://example.com"}
         with self.assertRaises(MissingField):
-            TarballPackage.from_dict(dict_missing_name_should_fail_on_contract)
-
-        # check RegistryPackageMetadata - it is used in TarballUnpinnedPackage
-        dct = {
-            "name": a.package,
-            "packages": [],  # note: required by RegistryPackageMetadata
-            "downloads": {"tarball": a_pinned.tarball},
-        }
-
-        metastore = RegistryPackageMetadata.from_dict(dct)
-        self.assertEqual(metastore.downloads.tarball, "http://example.com")
+            TarballPackage.from_dict(a_contract)
 
 
 class TestGitPackage(unittest.TestCase):