Repository: run-llama/llama_index
Pull request: [FIX] download_llama_pack for python packages containing multiple packs
# Description

- Previous download_llama_pack logic didn't traverse the Github tree deep enough, which posed a problem for packages with multiple packs (i.e., since there are more folders containing source code that needed to be discovered)
- This PR addresses that: uses the same Github trees API as before but traverses the tree recursively until there are no longer any "directories" to traverse.

Fixes #10777

## Type of Change

Please delete options that are not relevant.

- [x] Bug fix (non-breaking change which fixes an issue)

# How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

- [x] Ran download of `ChainOfTablePack` as well as `RagEvaluatorPack` and ensured that both worked
- [x] I stared at the code and made sure it makes sense


Problem statement:
Not able to use LLama Packs [Bug]: 
### Bug Description

I always get this error for any pack i download
FileNotFoundError: [Errno 2] No such file or directory: '/content/chain_of_table_pack/llama_index/packs/tables/base.py'

### Version

0.10.5

### Steps to Reproduce


download_llama_pack(
    "ChainOfTablePack",
    "./chain_of_table_pack",
    )

### Relevant Logs/Tracbacks

```shell
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-da45d7f3ade5> in <cell line: 3>()
      1 from llama_index.core.llama_pack import download_llama_pack
      2 
----> 3 download_llama_pack(
      4     "ChainOfTablePack",
      5     "./chain_of_table_pack",

4 frames
/usr/local/lib/python3.10/dist-packages/llama_index/core/llama_pack/download.py in download_llama_pack(llama_pack_class, download_dir, llama_pack_url, refresh_cache)
     54         )
     55     else:
---> 56         pack_cls = download_llama_pack_template(
     57             new_install_parent=new_install_parent,
     58             llama_pack_class=llama_pack_class,

/usr/local/lib/python3.10/dist-packages/llama_index/core/download/pack.py in download_llama_pack_template(new_install_parent, llama_pack_class, llama_pack_url, llama_pack_source_files_dir_path, refresh_cache, custom_dir, custom_path, base_file_name)
    124 
    125     module = util.module_from_spec(spec)
--> 126     spec.loader.exec_module(module)  # type: ignore
    127 
    128     return getattr(module, llama_pack_class)

/usr/lib/python3.10/importlib/_bootstrap_external.py in exec_module(self, module)

/usr/lib/python3.10/importlib/_bootstrap_external.py in get_code(self, fullname)

/usr/lib/python3.10/importlib/_bootstrap_external.py in get_data(self, path)

FileNotFoundError: [Errno 2] No such file or directory:
```

Hints from issue discussion:
_🤖_
<!-- Greeting -->
Hey @jaideep11061982, back again diving into the deep end with Llama Packs, I see! Let's tackle this head-on.

<!-- Answer -->
Based on the error logs you've provided, it seems like the issue is related to the file path of the pack you're trying to download. The error `FileNotFoundError: [Errno 2] No such file or directory: '/content/chain_of_table_pack/llama_index/packs/tables/base.py'` indicates that the file or directory you're trying to access doesn't exist.

This could be due to a couple of reasons:

1. The directory you're trying to download the pack into does not exist. In the `handle_download_llama_pack` function, the `download_dir` argument is used as the directory to download the pack into. If this directory does not exist, a FileNotFoundError could be raised. 

```py
def handle_download_llama_pack(
    llama_pack_class: Optional[str] = None,
    download_dir: Optional[str] = None,
    llama_pack_url: str = LLAMA_PACKS_CONTENTS_URL,
    **kwargs: Any,
) -> None:
    assert llama_pack_class is not None
    assert download_dir is not None

    download_llama_pack(
        llama_pack_class=llama_pack_class,
        download_dir=download_dir or "./custom_llama_pack",
        llama_pack_url=llama_pack_url,
    )
    print(f"Successfully downloaded {llama_pack_class} to {download_dir}")
```

2. The pack you're trying to download does not exist. In the `download_llama_pack` function, the `llama_pack_class` argument is used to specify the pack to download. If this pack does not exist, a FileNotFoundError could be raised.

To resolve this issue, ensure that the directory you're trying to download the pack into exists and that the pack you're trying to download also exists. You can refer to the [llama-index-cli/llama_index/cli/command_line.py](https://github.com/run-llama/llama_index/blob/main/llama-index-cli/llama_index/cli/command_line.py) for more details.

Also, there was a similar issue reported in the past, you can refer to this [issue](https://github.com/run-llama/llama_index/issues/945) for more context.

Regarding the structure of the directory where the Llama Packs are supposed to be downloaded, it is not explicitly mentioned in the provided context. However, the directory can be specified by the user when downloading a Llama Pack. This is done using the `--download-dir` argument in the CLI command or the second argument in the `download_llama_pack` function in Python.

CLI command:
```bash
llamaindex-cli download-llamapack <pack_name> --download-dir <pack_directory>
```

Python code:
```python
from llama_index.core.llama_pack import download_llama_pack

# download and install dependencies
pack_cls = download_llama_pack("<pack_name>", "<pack_directory>")
```

In both cases, `<pack_directory>` should be replaced with the desired directory path. You can refer to the [docs/community/llama_packs/root.md](https://github.com/run-llama/llama_index/blob/main/docs/community/llama_packs/root.md) for more details.

I hope this helps! If you have any more questions or need further clarification, feel free to ask.

## Sources
- [llama_index/setup_llama.py", line 7, in <module>     with open(Path(__file__).absolute().parents[0] / "llama_index" / "VERSION") as _f: FileNotFoundError: [Errno 2] No such file or directory:](https://github.com/run-llama/llama_index/issues/945)
- [llama-index-cli/llama_index/cli/command_line.py](https://github.com/run-llama/llama_index/blob/main/llama-index-cli/llama_index/cli/command_line.py)
- [docs/community/llama_packs/root.md](https://github.com/run-llama/llama_index/blob/main/docs/community/llama_packs/root.md)

<!-- Dosu Comment Footer -->
<details open>
<summary>

#### About Dosu
</summary>

This response is meant to be useful and save you time. It is **not meant to be a precise solution**, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.
</details>
Thanks @jaideep11061982 for raising. Can confirm that I am able to reproduce the bug. Looks like this is caused by the logic for downloading packs not taking into account of multi-level code structure. Specifically, its looking for the `base.py` file, but for [this](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-tables/llama_index/packs/tables) pack, the "base.py" files are found one level deeper in the source code.

Will submit a fix for this shortly.
@jaideep11061982 In the meantime, if you're intending to only use the pack and not looking to customize it, you can directly download via pip as it is its own pypi package:

`pip install llama-index-packs-tables`

Then in python
```
from llama_index.packs.tables import ChainOfTablePack

# and you can use ChainOfTablePack
```
@jaideep11061982 can you please tell me which other packs you've tested? This issue should only occur with packs that come from packages consisting of multiple packs.

In v0.10 llama-packs are packaged in Python packages, where a single Python package can, though not common, consist of multiple llama-packs. It looks like there are only 3 such packages (out of ~50 packages). These 3 packages along with the llama-packs defined in them (6 total across the three packages) are listed below.

- `HybridFusionRetrieverPack`, `QueryRewritingRetrieverPack` (come from [llama-index-packs-fusion-retriever](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-fusion-retriever/llama_index/packs/fusion_retriever) package)
- `EmbeddedTablesUnstructuredRetrieverPack`, `RecursiveRetrieverSmallToBigPack` (come from [llama-index-packs-recursive-retriever](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-recursive-retriever/llama_index/packs/recursive_retriever) package)
- `ChainOfTablePack`, `MixSelfConsistencyPack` (come from [llama-index-packs-tables](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-tables/llama_index/packs/tables) package)

Were you only testing download_llama_pack on these 6 LlamaPack classes?
> @jaideep11061982 can you please tell me which other packs you've tested? This issue should only occur with packs that come from packages consisting of multiple packs.
> 
> In v0.10 llama-packs are packaged in Python packages, where a single Python package can, though not common, consist of multiple llama-packs. It looks like there are only 3 such packages (out of ~50 packages). These 3 packages along with the llama-packs defined in them (6 total across the three packages) are listed below.
> 
> * `HybridFusionRetrieverPack`, `QueryRewritingRetrieverPack` (come from [llama-index-packs-fusion-retriever](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-fusion-retriever/llama_index/packs/fusion_retriever) package)
> * `EmbeddedTablesUnstructuredRetrieverPack`, `RecursiveRetrieverSmallToBigPack` (come from [llama-index-packs-recursive-retriever](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-recursive-retriever/llama_index/packs/recursive_retriever) package)
> * `ChainOfTablePack`, `MixSelfConsistencyPack` (come from [llama-index-packs-tables](https://github.com/run-llama/llama_index/tree/main/llama-index-packs/llama-index-packs-tables/llama_index/packs/tables) package)
> 
> Were you only testing download_llama_pack on these 6 LlamaPack classes?

It happens for voyage one also "VoyageQueryEnginePack", "./voyage_pack" @nerdai 
Hey @jaideep11061982:

I just checked `VoyageQueryEnginePack` and was able to download this just fine. Can you try again? Are you using a fresh environment?

```python
from llama_index.core.llama_pack import download_llama_pack

VoyageQueryEnginePack = download_llama_pack("VoyageQueryEnginePack", "./voyage_pack")
```

I'm running llama-index v0.10.5.
i just tested out 0.10.3 it wasnt working.
Moreover  can your earlier method work always ?
from llama_index.packs.tables import ChainOfTablePack
Oh, it should be working as of 0.10.0 I think.

The method of pip installing should always work unless there is a bug contained in the source code preventing it from importing -- though I believe we did conduct tests to ensure that there were no import errors on almost all of our integration packages.

Ultimately:

- Use `pip install <llama-pack-package>` if you just want to utilize the pack (probably most our users should do this?)
- Use `download_llama_pack` if you want to use the pack and also customize it (i.e., change the source code)
importing chainoftable pack fails with error 
`cannot import name 'version_short' from 'pydantic.version' (/opt/conda/lib/python3.10/site-`packages/pydantic/version.cpython-310-x86_64-linux-gnu.so)
from ChainOfTablePack import ChainOfTableQueryEngine, serialize_table #fails

from chain_of_table_pack import ChainOfTableQueryEngine, serialize_table   this will work with download one. 
How to make these work, I am trying the sample notebook on this https://github.com/run-llama/llama-hub/blob/main/llama_hub/llama_packs/tables/chain_of_table/chain_of_table.ipynb
Hi @jaideep11061982, I just tried importing both `ChainOfTableQueryEngine` and `serialize_table` and it worked for me.

Are you sure that you are using a fresh virtual environment? It looks like you're using conda, please try this:

```sh
# conda
conda create -n venv python=3.11 anaconda # can use any Python 3.9 - 3.12 
conda activate venv
pip install llama-index llama-index-packs-tables
```

And with `venv` active:
```python
from llama_index.packs.tables.chain_of_table.base import ChainOfTableQueryEngine, serialize_table

# these should work
# in future we can add seralize_table as top-level import but for now the above should work
```

Commit message: remove branch and repo constants

Patch to review:
diff --git a/llama-index-core/llama_index/core/download/pack.py b/llama-index-core/llama_index/core/download/pack.py
index bcf7430470..f32d43ab05 100644
--- a/llama-index-core/llama_index/core/download/pack.py
+++ b/llama-index-core/llama_index/core/download/pack.py
@@ -13,17 +13,15 @@
 from llama_index.core.download.utils import (
     ChangeDirectory,
     get_file_content,
-    get_source_files_list,
     initialize_directory,
+    get_source_files_recursive,
 )
 
-BRANCH = "nerdai/migration-v0_10_0"
-REPO = "nerdai"
 LLAMA_PACKS_CONTENTS_URL = (
-    f"https://raw.githubusercontent.com/{REPO}/llama_index/{BRANCH}/llama-index-packs"
+    f"https://raw.githubusercontent.com/run-llama/llama_index/main/llama-index-packs"
 )
 LLAMA_PACKS_SOURCE_FILES_GITHUB_TREE_URL = (
-    f"https://github.com/{REPO}/llama_index/tree/{BRANCH}/llama-index-packs"
+    f"https://github.com/run-llama/llama_index/tree/main"
 )
 PY_NAMESPACE = "llama_index/packs"
 
@@ -51,17 +49,24 @@ def download_module_and_reqs(
         os.makedirs(module_path, exist_ok=True)
 
         # download all source files
-        source_files = get_source_files_list(
+        source_files = get_source_files_recursive(
             str(remote_source_dir_path),
-            f"/{package}/{PY_NAMESPACE}/{sub_module}",
+            f"/llama-index-packs/{package}/{PY_NAMESPACE}/{sub_module}",
         )
 
         for source_file in source_files:
             source_file_raw_content, _ = get_file_content(
                 str(remote_dir_path),
-                f"/{package}/{PY_NAMESPACE}/{sub_module}/{source_file}",
+                f"{source_file}",
             )
-            with open(f"{module_path}/{source_file}", "w") as f:
+            local_source_file_path = (
+                f"{local_dir_path}/{'/'.join(source_file.split('/')[2:])}"
+            )
+            # ensure parent dir of file exists
+            Path(local_source_file_path).parent.absolute().mkdir(
+                parents=True, exist_ok=True
+            )
+            with open(local_source_file_path, "w") as f:
                 f.write(source_file_raw_content)
 
     # pyproject.toml and README
@@ -99,7 +104,7 @@ def download_llama_pack_template(
     refresh_cache: bool = False,
     custom_dir: Optional[str] = None,
     custom_path: Optional[str] = None,
-    base_file_name: str = "base.py",
+    base_file_name: str = "__init__.py",
 ) -> Any:
     # create directory / get path
     dirpath = initialize_directory(custom_path=custom_path, custom_dir=custom_dir)
diff --git a/llama-index-core/llama_index/core/download/utils.py b/llama-index-core/llama_index/core/download/utils.py
index 933e2ebd18..e8dd91c284 100644
--- a/llama-index-core/llama_index/core/download/utils.py
+++ b/llama-index-core/llama_index/core/download/utils.py
@@ -95,6 +95,47 @@ def get_source_files_list(source_tree_url: str, path: str) -> List[str]:
     return [item["name"] for item in payload["tree"]["items"]]
 
 
+def recursive_tree_traverse(
+    tree_urls: List[Tuple[str, str]], acc: List[str], source_tree_url: str
+):
+    """Recursively traversge Github trees to get all file paths in a folder."""
+    if not tree_urls:
+        return acc
+    else:
+        url = tree_urls[0]
+
+        try:
+            res = requests.get(url)
+            tree_elements = res.json()["payload"]["tree"]["items"]
+        except Exception:
+            raise ValueError("Failed to traverse github tree source.")
+
+        new_trees = [
+            source_tree_url + "/" + el["path"]
+            for el in tree_elements
+            if el["contentType"] == "directory"
+        ]
+
+        acc += [
+            el["path"].replace("llama-index-packs/", "/")
+            for el in tree_elements
+            if el["contentType"] == "file"
+        ]
+
+        return recursive_tree_traverse(
+            tree_urls=tree_urls[1:] + new_trees,
+            acc=acc,
+            source_tree_url=source_tree_url,
+        )
+
+
+def get_source_files_recursive(source_tree_url: str, path: str) -> List[str]:
+    """Get source files of a Github folder recursively."""
+    initial_url = source_tree_url + path + "?recursive=1"
+    initial_tree_urls = [initial_url]
+    return recursive_tree_traverse(initial_tree_urls, [], source_tree_url)
+
+
 class ChangeDirectory:
     """Context manager for changing the current working directory."""
 


Code review comments:
diff_hunk: @@ -13,17 +13,15 @@
 from llama_index.core.download.utils import (
     ChangeDirectory,
     get_file_content,
-    get_source_files_list,
     initialize_directory,
+    get_source_files_recursive,
 )
 
-BRANCH = "nerdai/migration-v0_10_0"
-REPO = "nerdai"
 LLAMA_PACKS_CONTENTS_URL = (
-    f"https://raw.githubusercontent.com/{REPO}/llama_index/{BRANCH}/llama-index-packs"
+    f"https://raw.githubusercontent.com/run-llama/llama_index/main/llama-index-packs"
 )
 LLAMA_PACKS_SOURCE_FILES_GITHUB_TREE_URL = (
-    f"https://github.com/{REPO}/llama_index/tree/{BRANCH}/llama-index-packs"
+    f"https://github.com/run-llama/llama_index/tree/main"
path: llama-index-core/llama_index/core/download/pack.py
text: @user1:
We can remove the f-string format too I guess if we are not using any constants