Repository: bentoml/BentoML

Issue:
feature: allow nested `requirements.txt` files 
### Feature request

I would like to be able to have a `requirements.txt` file which references other requirements files with `-r`.

### Motivation

I would like to be able to you `pip-compile-multi` to manage the dependencies of my models. Often there are different dependencies at serve and train time with some overlap. Therefore, I'd like to be able to define the common dependencies in one file, the serving specific ones in another file, and the training specific ones in a third file.

ie.

**model.txt**
```
scikit-learn
```

**train.txt**
```
-r model.txt
pandas
```

**serve.txt**
```
-r model.txt
pydantic
```



### Other

_No response_


Discussion:

Resolving patch:
diff --git a/src/bentoml/_internal/bento/build_config.py b/src/bentoml/_internal/bento/build_config.py
--- a/src/bentoml/_internal/bento/build_config.py
+++ b/src/bentoml/_internal/bento/build_config.py
@@ -17,6 +17,7 @@
 import fs.copy
 from dotenv import dotenv_values  # type: ignore
 from pathspec import PathSpec
+from pip_requirements_parser import RequirementsFile
 
 from .gen import generate_dockerfile
 from ..utils import bentoml_cattr
@@ -544,15 +545,22 @@ def write_to_bento(self, bento_fs: FS, build_ctx: str) -> None:
             f.write(install_sh)
 
         if self.requirements_txt is not None:
-            requirements_txt_file = resolve_user_filepath(
-                self.requirements_txt, build_ctx
-            )
-            copy_file_to_fs_folder(
-                requirements_txt_file,
-                bento_fs,
-                py_folder,
-                dst_filename="requirements.txt",
+            requirements_txt = RequirementsFile.from_file(
+                resolve_user_filepath(self.requirements_txt, build_ctx),
+                include_nested=True,
             )
+            # We need to make sure that we don't write any file references
+            # back into the final `requirements.txt` file. We've already
+            # resolved them and included their contents so we can discard
+            # them.
+            for option_line in requirements_txt.options:
+                option_line.options.pop("constraints", None)
+                option_line.options.pop("requirements", None)
+
+            with bento_fs.open(
+                fs.path.combine(py_folder, "requirements.txt"), "w"
+            ) as f:
+                f.write(requirements_txt.dumps(preserve_one_empty_line=True))
         elif self.packages is not None:
             with bento_fs.open(fs.path.join(py_folder, "requirements.txt"), "w") as f:
                 f.write("\n".join(self.packages))
diff --git a/src/bentoml/_internal/utils/__init__.py b/src/bentoml/_internal/utils/__init__.py
--- a/src/bentoml/_internal/utils/__init__.py
+++ b/src/bentoml/_internal/utils/__init__.py
@@ -279,6 +279,7 @@ def copy_file_to_fs_folder(
     src_fs = fs.open_fs(dir_name)
     dst_filename = file_name if dst_filename is None else dst_filename
     dst_path = fs.path.join(dst_folder_path, dst_filename)
+    dst_fs.makedir(dst_folder_path, recreate=True)
     fs.copy.copy_file(src_fs, file_name, dst_fs, dst_path)
 
 


Test patch:
diff --git a/tests/e2e/bento_server_http/bentofile.yaml b/tests/e2e/bento_server_http/bentofile.yaml
--- a/tests/e2e/bento_server_http/bentofile.yaml
+++ b/tests/e2e/bento_server_http/bentofile.yaml
@@ -1,6 +1,6 @@
 service: service:svc
 python:
-  requirements_txt: ./requirements.txt
+  requirements_txt: ./requirements/service.txt
 exclude:
   - pickle_model.py
   - train.py
diff --git a/tests/e2e/bento_server_http/requirements/service.txt b/tests/e2e/bento_server_http/requirements/service.txt
new file mode 100644
--- /dev/null
+++ b/tests/e2e/bento_server_http/requirements/service.txt
@@ -0,0 +1,5 @@
+-r train/requirements.txt
+
+pydantic
+Pillow
+fastapi
diff --git a/tests/e2e/bento_server_http/requirements.txt b/tests/e2e/bento_server_http/requirements/train/requirements.txt
similarity index 53%
rename from tests/e2e/bento_server_http/requirements.txt
rename to tests/e2e/bento_server_http/requirements/train/requirements.txt
--- a/tests/e2e/bento_server_http/requirements.txt
+++ b/tests/e2e/bento_server_http/requirements/train/requirements.txt
@@ -1,6 +1,3 @@
 pandas
-pydantic
-Pillow
 scikit-learn
 pyarrow
-fastapi