Repository: conan-io/conan

Issue:
Deployment generators to export package contents to a specific layout
Coming from #3243 and related to #4827 and the default deployable concept in #3280.

I have played witht the idea of having a generator that deploys the files of a dependency tree to a specific layout. They idea is, instead of using the paths to generate a file, use them to copy the needed files to the installation folder.

In order to achive this you would need to create a "dumb recipe" (*conanfile.txt) with the requirements of the package you want to deploy ("ncurses" in this case)
and the custom generator (called "deployer" in this case).

*conanfile.txt*
```
[requires]
ncurses/6.1@danimtb/testing

deployer/0.1@danimtb/testing

[generators]
deployer
```

The custom generator I created as a example is the following:

*conanfile.py*
```
import os
import shutil

from conans.model import Generator
from conans import ConanFile


class deployer(Generator):
    @property
    def filename(self):
        return "conandeployer.txt"

    @property
    def content(self):
        for dep_name, dep_cpp_info in self.deps_build_info.dependencies:
            if dep_name == "deployer":
                continue
            for key in ["include_paths", "lib_paths", "bin_paths"]:
                if getattr(dep_cpp_info, key):
                    for path in getattr(dep_cpp_info, key):
                        if os.listdir(path):
                            src = os.path.normpath(path)
                            dir_name = os.path.basename(path)
                            dst = os.path.join(self.output_path, dep_name, dir_name)
                            shutil.copytree(src, dst)
        return "Files deployed!"


class MyCustomGeneratorPackage(ConanFile):
    name = "deployer"
    version = "0.1"
```
There are some tricks to make the generator work, for example using the ``self.output_path`` attribute.

However, this approach can be applied externally without modifying the recipe of the package to be deployed.

Ideas that this issue tries to merge:
- "Deploy packages to a custom structure (like *chroot*)" by @brianthelion in #3243 
- "Deploy packages to a custom structure using hooks" by @brianthelion in #4827
- "Having a default deploy method in recipes" by @solvingj in  #3280
- "Using a proxy package with a deploy method" by @monsdar in  #3280 (this idea is pretty similar to the one exposed here).


Discussion:
After discussing the  use case for deploy, we'd go forward with the idea of the deploy generator.

Goal:
Create a builtin generator “deploy” that will copy the contents with the following structure:

```
install_folder/libA/include
install_folder/libA/bin
…
install_folder/libB/include
install_folder/libB/bin
```
The “output_folder” and the way to create a custom one should be documented also.

Look at the "deploy" project to review how many of them could be closed.
Additional requirement: process ALL requirements ;)
Means: everything in the graph has to be deployed.
@Aalmann sure, that is the idea. Everything available through the [deps_cpp_info](https://docs.conan.io/en/latest/reference/conanfile/attributes.html#deps-cpp-info) object
I also need a feature where I can deploy a tool e.g. to ~/.local/bin and all its dependent shared libraries to ~/.local/lib. Additionally patching the runpath (as shown with readelf -d mylib.so) should be possible that tells a lib where to search for dependencies at runtime.
Here's my hook-based PoC: https://gist.github.com/brianthelion/180020d161a0c76cc0e6e9ddf350ad59

My approach makes a key assumption that the generator approach does not, namely, that the deployment environment must be built and managed incrementally. The code above enacts the deployment policy at each `conan install`. It works quite well.

One slight advantage here is that the user of the hook doesn't have to *do* anything other than install the hook. With a generator, they need to both (1) install the generator and (2) write a `conanfile` to enable the generator.
@brianthelion thanks for sharing your approach with hooks, I see it has some advantages compared to the generator.

However, I guess you would have to activate the hook only for `conan install` commands. Otherwise the deployment will also get triggered when doing a `conan create` or `conan package`. Also, you are deploying every package that gets installed, private and build requirements included, which might be not needed in the final deployment.

Nevertheless, I see the value of the hook and I like the idea of having this behavior isolated there. Still I think the generator approach is simpler and integrates nicely with the role of the generators
@danimtb Agree with and appreciate your comments. The hook-based approach and the generator-based approach seem to address different-but-similar use cases.

> However, I guess you would have to activate the hook only for conan install commands. Otherwise the deployment will also get triggered when doing a `conan create` or `conan package`. Also, you are deploying every package that gets installed, private and build requirements included, which might be not needed in the final deployment.

You're correct here. It's not clear that the current implementation of hooks allows us to isolate it strictly to `conan install` events. I think the hooks API could use some embellishment is this regard. It's also not clear how to get hooks to run on `conan remove` with the current API. But this is a digression from the current thread. Should we continue that discussion under a new issue?

Resolving patch:
diff --git a/conans/client/generators/__init__.py b/conans/client/generators/__init__.py
--- a/conans/client/generators/__init__.py
+++ b/conans/client/generators/__init__.py
@@ -13,6 +13,7 @@
 from .cmake import CMakeGenerator
 from .cmake_multi import CMakeMultiGenerator
 from .cmake_paths import CMakePathsGenerator
+from .deploy import DeployGenerator
 from .gcc import GCCGenerator
 from .json_generator import JsonGenerator
 from .make import MakeGenerator
@@ -77,6 +78,7 @@ def __getitem__(self, key):
 registered_generators.add("b2", B2Generator)
 registered_generators.add("premake", PremakeGenerator)
 registered_generators.add("make", MakeGenerator)
+registered_generators.add("deploy", DeployGenerator)
 
 
 def write_generators(conanfile, path, output):
diff --git a/conans/client/generators/deploy.py b/conans/client/generators/deploy.py
new file mode 100644
--- /dev/null
+++ b/conans/client/generators/deploy.py
@@ -0,0 +1,47 @@
+import calendar
+import os
+import shutil
+import time
+
+from conans.model import Generator
+from conans import ConanFile
+from conans.model.manifest import FileTreeManifest
+from conans.paths import BUILD_INFO_DEPLOY
+from conans.util.files import mkdir, md5sum
+
+FILTERED_FILES = ["conaninfo.txt", "conanmanifest.txt"]
+
+
+class DeployGenerator(Generator):
+
+    def deploy_manifest_content(self, copied_files):
+        date = calendar.timegm(time.gmtime())
+        file_dict = {}
+        for f in copied_files:
+            abs_path = os.path.join(self.output_path, f)
+            file_dict[f] = md5sum(abs_path)
+        manifest = FileTreeManifest(date, file_dict)
+        return repr(manifest)
+
+    @property
+    def filename(self):
+        return BUILD_INFO_DEPLOY
+
+    @property
+    def content(self):
+        copied_files = []
+
+        for dep_name in self.conanfile.deps_cpp_info.deps:
+            rootpath = self.conanfile.deps_cpp_info[dep_name].rootpath
+            for root, _, files in os.walk(os.path.normpath(rootpath)):
+                for f in files:
+                    if f in FILTERED_FILES:
+                        continue
+                    src = os.path.normpath(os.path.join(root, f))
+                    dst = os.path.join(self.output_path, dep_name,
+                                       os.path.relpath(root, rootpath), f)
+                    dst = os.path.normpath(dst)
+                    mkdir(os.path.dirname(dst))
+                    shutil.copyfile(src, dst)
+                    copied_files.append(dst)
+        return self.deploy_manifest_content(copied_files)
diff --git a/conans/paths/__init__.py b/conans/paths/__init__.py
--- a/conans/paths/__init__.py
+++ b/conans/paths/__init__.py
@@ -36,6 +36,7 @@ def get_conan_user_home():
 BUILD_INFO_XCODE = 'conanbuildinfo.xcconfig'
 BUILD_INFO_PREMAKE = 'conanbuildinfo.premake.lua'
 BUILD_INFO_MAKE = 'conanbuildinfo.mak'
+BUILD_INFO_DEPLOY = 'deploy_manifest.txt'
 CONANINFO = "conaninfo.txt"
 CONANENV = "conanenv.txt"
 SYSTEM_REQS = "system_reqs.txt"


Test patch:
diff --git a/conans/test/functional/generators/deploy_test.py b/conans/test/functional/generators/deploy_test.py
new file mode 100644
--- /dev/null
+++ b/conans/test/functional/generators/deploy_test.py
@@ -0,0 +1,108 @@
+import os
+import unittest
+
+from conans import load
+from conans.model.ref import ConanFileReference
+from conans.test.utils.test_files import temp_folder
+from conans.test.utils.tools import GenConanfile, TurboTestClient
+
+
+class DeployGeneratorTest(unittest.TestCase):
+    """
+    Deploy generator set of tests with only one package
+    """
+
+    def setUp(self):
+        conanfile = GenConanfile()
+        conanfile.with_package_file("include/header.h", "whatever")
+        conanfile.with_package_file("my_libs/file.lib", "whatever")
+        conanfile.with_package_file("file.config", "whatever")
+
+        self.client = TurboTestClient()
+        ref = ConanFileReference("name", "version", "user", "channel")
+        self.client.create(ref, conanfile)
+        self.client.current_folder = temp_folder()
+        self.client.run("install %s -g deploy" % ref.full_repr())
+
+    def deploy_folder_path_test(self):
+        base_path = os.path.join(self.client.current_folder, "name")
+        expected_include_path = os.path.join(base_path, "include")
+        expected_lib_path = os.path.join(base_path, "my_libs")
+        self.assertTrue(os.path.exists(expected_include_path))
+        self.assertTrue(os.path.exists(expected_lib_path))
+
+    def deploy_manifest_path_test(self):
+        expected_manifest_path = os.path.join(self.client.current_folder, "deploy_manifest.txt")
+        self.assertTrue(os.path.exists(expected_manifest_path))
+
+    def deploy_manifest_content_test(self):
+        base_path = os.path.join(self.client.current_folder, "name")
+        header_path = os.path.join(base_path, "include", "header.h")
+        lib_path = os.path.join(base_path, "my_libs", "file.lib")
+        config_path = os.path.join(base_path, "file.config")
+        manifest_path = os.path.join(self.client.current_folder, "deploy_manifest.txt")
+        content = load(manifest_path)
+        self.assertIn(header_path, content)
+        self.assertIn(lib_path, content)
+        self.assertIn(config_path, content)
+
+    def no_conan_metadata_files_test(self):
+        metadata_files = ["conaninfo.txt", "conanmanifest.txt"]
+        # Assert not in directory tree
+        for root, _, _ in os.walk(self.client.current_folder):
+            for name in metadata_files:
+                self.assertNotIn(name, os.listdir(root))
+        # Assert not in manifest
+        manifest_path = os.path.join(self.client.current_folder, "deploy_manifest.txt")
+        content = load(manifest_path)
+        for name in metadata_files:
+            self.assertNotIn(name, content)
+
+
+class DeployGeneratorGraphTest(unittest.TestCase):
+    """
+    Deploy generator set of tests with more than one package in the graph
+    """
+
+    def setUp(self):
+        conanfile1 = GenConanfile()
+        conanfile1.with_package_file("include/header1.h", "whatever")
+        conanfile1.with_package_file("my_libs/file1.lib", "whatever")
+        conanfile1.with_package_file("file1.config", "whatever")
+        ref1 = ConanFileReference("name1", "version", "user", "channel")
+
+        conanfile2 = GenConanfile()
+        conanfile2.with_requirement(ref1)
+        conanfile2.with_package_file("include_files/header2.h", "whatever")
+        conanfile2.with_package_file("my_other_libs/file2.lib", "whatever")
+        conanfile2.with_package_file("build/file2.config", "whatever")
+        ref2 = ConanFileReference("name2", "version", "user", "channel")
+
+        self.client = TurboTestClient()
+        self.client.create(ref1, conanfile1)
+        self.client.create(ref2, conanfile2)
+        self.client.current_folder = temp_folder()
+        self.client.run("install %s -g deploy" % ref2.full_repr())
+
+    def get_expected_paths(self):
+        base1_path = os.path.join(self.client.current_folder, "name1")
+        header1_path = os.path.join(base1_path, "include", "header1.h")
+        lib1_path = os.path.join(base1_path, "my_libs", "file1.lib")
+        config1_path = os.path.join(base1_path, "file1.config")
+
+        base2_path = os.path.join(self.client.current_folder, "name2")
+        header2_path = os.path.join(base2_path, "include_files", "header2.h")
+        lib2_path = os.path.join(base2_path, "my_other_libs", "file2.lib")
+        config2_path = os.path.join(base2_path, "build", "file2.config")
+
+        return [header1_path, lib1_path, config1_path, header2_path, lib2_path, config2_path]
+
+    def deploy_manifest_content_test(self):
+        manifest_path = os.path.join(self.client.current_folder, "deploy_manifest.txt")
+        content = load(manifest_path)
+        for path in self.get_expected_paths():
+            self.assertIn(path, content)
+
+    def file_paths_test(self):
+        for path in self.get_expected_paths():
+            self.assertTrue(os.path.exists(path))
diff --git a/conans/test/utils/tools.py b/conans/test/utils/tools.py
--- a/conans/test/utils/tools.py
+++ b/conans/test/utils/tools.py
@@ -993,8 +993,8 @@ class GenConanfile(object):
         with_default_option("shared", True).\
         with_build_msg("holaaa").\
         with_build_msg("adiooos").\
-        with_package_file("file.txt", "hola"). \
-        with_package_file("file2.txt", "hola").gen()
+        with_package_file("file.txt", "hola").\
+        with_package_file("file2.txt", "hola")
     """
 
     def __init__(self):
@@ -1041,11 +1041,11 @@ def with_default_option(self, option_name, value):
     def with_package_file(self, file_name, contents=None, env_var=None):
         if not contents and not env_var:
             raise Exception("Specify contents or env_var")
+        self.with_import("import os")
         self.with_import("from conans import tools")
         if contents:
             self._package_files[file_name] = contents
         if env_var:
-            self.with_import("import os")
             self._package_files_env[file_name] = env_var
         return self