Repository: conan-io/conan

Issue:
[feature] Allow user-defined generators to be used with conan install -g
The ability to "mix in" generators when installing a package reference is a unique feature that I think could be improved by allowing the use of user-defined generators.

The motivation for this comes from wanting to write a custom generator which would take any existing package and generate new file(s) from it at install time, similar to some of the built-in generators. Since this is restricted to built-in generators only, a new recipe would need to be created, maintained, and kept around for every package I wanted to use my custom generator with.

I have a very simple implementation of this feature [here](https://github.com/cobalt77/conan/commit/e1947d58a7f9da78d6ddf24cdd81dca911cb2170). I added a new folder in the client cache called 'generators' which can contain user-defined generators. These can be installed using 'conan config install'. When 'conan install' is invoked with a package reference, user-defined generators from the generators folder are loaded and available to use with -g.

- [x] I've read the [CONTRIBUTING guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).



Discussion:
I agree with this. There is some pending improvements and discussions to be done around generators, see https://github.com/conan-io/conan/issues/6790

Having generators as config in the conan cache would make sense. Although a bit earlier than expected, if you want please do a PR with your implementation and lets discuss about it in next Conan 1.29, we need to think about other implications so no guarantee that could be merged, but lets look into it. Thanks!
I have the same concern as stated [here](https://github.com/conan-io/conan/issues/6790#issuecomment-647960865) by @maikelvdh: how to handle it properly? how to record the version used? (reproducibility and traceability)

A feature I think about from time to time (but never too seriously) is adding `python_requires` to the profiles, Conan would add those python-requires to every package, functions would be available, **generators would be registered**, other tools like build-helpers or toolchains classes,... and that information would be recorded in the lockfile.
@jgsogo The main motivation for this comes from wanting to using generators at install time to deploy packages in specific ways without the packages themselves needing to know about the generator. For instance, my company's use case is to use a generator to transform a conan package into an RPM. Managing dozens of in-house libraries using individual RPMs is a **nightmare** (I cannot stress this enough), but with Conan packages it is trivial. The missing piece is being able to convert any of our conan packages into an RPM when necessary.

My implementation has the effect of profile python_requires for generators only. these generators would be internal - created by a developer for their team to integrate the package with their own environment, just like profiles. The generators are not versioned and have no recipe (just like the builtin ones); when using them at install time, the generator simply interacts with the package using a virtual recipe and the package does not need to know about it or be able to reproduce the installation.

This design allows for a single package to be deployed in any number of ways by different generators. You could set up generators to transform a package into a variety of formats which could make deploying artifacts a lot less tedious.

I don't think this is a perfect solution, but it is a solid starting point for a feature that does not have a good workaround as of now. I have some ideas on how this can be improved further (but would require non-trivial refactoring). The end goal in my mind would be to separate generators into two categories - those which a _package uses_, and those which _use packages_. 
Yes, I totally understand your motivation, and I really like your last statement:

> The end goal in my mind would be to separate generators into two categories - those which a package uses, and those which use packages.

💯 🙌 

As this is a "look into", it is good to consider all the alternatives and think about the mid-term and the final approach we want. In my opinion, I agree with you, Conan would benefit from the introduction of another category of items that _use packages_ (some current generators could be moved to that category, like `virtualxxxenv`, `deploy` or `markdown`). The main objective os these classes is to extract data/binaries from Conan and make it available to external tools, and Conan no longer consumes this information. I agree this kind of _generators_ doesn't need to be versioned.

Do these _generators_ make sense only when running `conan install <reference> -xxx <new-generator>`? Will they ever be used inside a recipe to interact with the requirements? 
I think they make the most sense when separated from the recipes themselves, however the recipes should be able to provide as much abstract information as possible to allow usage with many different \<new category> generators. This information might be things like installation folders, functions to run on pre/post install, etc. 

Resolving patch:
diff --git a/conans/client/cache/cache.py b/conans/client/cache/cache.py
--- a/conans/client/cache/cache.py
+++ b/conans/client/cache/cache.py
@@ -32,6 +32,7 @@
 PROFILES_FOLDER = "profiles"
 HOOKS_FOLDER = "hooks"
 TEMPLATES_FOLDER = "templates"
+GENERATORS_FOLDER = "generators"
 
 
 def is_case_insensitive_os():
@@ -167,6 +168,10 @@ def profiles_path(self):
     def settings_path(self):
         return join(self.cache_folder, CONAN_SETTINGS)
 
+    @property
+    def generators_path(self):
+        return join(self.cache_folder, GENERATORS_FOLDER)
+
     @property
     def default_profile_path(self):
         if os.path.isabs(self.config.default_profile):
@@ -208,6 +213,17 @@ def hooks(self):
                 hooks.append(hook_name[:-3])
         return hooks
 
+    @property
+    def generators(self):
+        """Returns a list of generator paths inside the generators folder"""
+        generators = []
+        if os.path.exists(self.generators_path):
+            for path in os.listdir(self.generators_path):
+                generator = os.path.join(self.generators_path, path)
+                if os.path.isfile(generator) and generator.endswith(".py"):
+                    generators.append(generator)
+        return generators
+
     def delete_empty_dirs(self, deleted_refs):
         for ref in deleted_refs:
             ref_path = self.package_layout(ref).base_folder()
diff --git a/conans/client/installer.py b/conans/client/installer.py
--- a/conans/client/installer.py
+++ b/conans/client/installer.py
@@ -292,6 +292,11 @@ def __init__(self, app, recorder):
         self._binaries_analyzer = app.binaries_analyzer
         self._hook_manager = app.hook_manager
         self._generator_manager = app.generator_manager
+        # Load custom generators from the cache, generators are part of the binary
+        # build and install. Generators loaded here from the cache will have precedence
+        # and overwrite possible generators loaded from packages (requires)
+        for generator_path in app.cache.generators:
+            app.loader.load_generators(generator_path)
 
     def install(self, deps_graph, remotes, build_mode, update, keep_build=False, graph_info=None):
         # order by levels and separate the root node (ref=None) from the rest
diff --git a/conans/client/loader.py b/conans/client/loader.py
--- a/conans/client/loader.py
+++ b/conans/client/loader.py
@@ -24,6 +24,7 @@
 
 
 class ConanFileLoader(object):
+
     def __init__(self, runner, output, python_requires, generator_manager=None, pyreq_loader=None):
         self._runner = runner
         self._generator_manager = generator_manager
@@ -87,6 +88,22 @@ def load_basic_module(self, conanfile_path, lock_python_requires=None, user=None
         except ConanException as e:
             raise ConanException("Error loading conanfile at '{}': {}".format(conanfile_path, e))
 
+    def load_generators(self, conanfile_path):
+        """ Load generator classes from a module. Any non-generator classes
+        will be ignored. python_requires is not processed.
+        """
+        """ Parses a python in-memory module and adds any generators found
+            to the provided generator list
+            @param conanfile_module: the module to be processed
+            """
+        conanfile_module, module_id = _parse_conanfile(conanfile_path)
+        for name, attr in conanfile_module.__dict__.items():
+            if (name.startswith("_") or not inspect.isclass(attr) or
+                    attr.__dict__.get("__module__") != module_id):
+                continue
+            if issubclass(attr, Generator) and attr != Generator:
+                self._generator_manager.add(attr.__name__, attr, custom=True)
+
     @staticmethod
     def _load_data(conanfile_path):
         data_path = os.path.join(os.path.dirname(conanfile_path), DATA_YML)


Test patch:
diff --git a/conans/test/functional/generators/generator_from_cache_test.py b/conans/test/functional/generators/generator_from_cache_test.py
new file mode 100644
--- /dev/null
+++ b/conans/test/functional/generators/generator_from_cache_test.py
@@ -0,0 +1,115 @@
+# -*- coding: utf-8 -*-
+
+import json
+import os
+import textwrap
+import unittest
+
+from conans.test.utils.tools import TestClient
+from conans.util.files import save
+
+
+class GeneratorFromCacheTest(unittest.TestCase):
+    """
+    Tests user defined generators loaded from the generators folder in
+    the client cache.
+    """
+
+    def install_cmdline_test(self):
+        """
+        Test the availability of user-defined generators from the command-line
+        using `install -g`
+        """
+        client = TestClient(cache_autopopulate=True)
+
+        # Set up generator. This generator will interact with another package.
+        test_generator_contents = textwrap.dedent("""
+            import json
+
+            from conans.model import Generator
+
+            class user_defined(Generator):
+                @property
+                def filename(self):
+                    return "userdefined.json"
+
+                @property
+                def content(self):
+                    return json.dumps(self.deps_env_info.vars)
+            """)
+
+        # Save generator to cache_folder/generators.
+        generator_path = os.path.join(client.cache.generators_path, "user_defined_generator.py")
+        save(generator_path, test_generator_contents)
+
+        conanfile_py = textwrap.dedent("""
+            from conans import ConanFile
+
+            class HelloConan(ConanFile):
+                build_policy = "always"
+
+                def package_info(self):
+                    self.env_info.MY_ENV_VAR1 = "foo"
+                    self.env_info.MY_ENV_VAR2 = "bar"
+                    self.env_info.MY_ENV_VAR3 = "baz"
+                    """)
+        client.save({"conanfile.py": conanfile_py})
+
+        # Test the install using a reference
+        client.run("export . Hello/0.1@lasote/testing")
+        client.run("install Hello/0.1@lasote/testing -g user_defined")
+
+        data = client.load("userdefined.json")
+        json_data = json.loads(data)
+        self.assertEqual(json_data["MY_ENV_VAR1"], "foo")
+        self.assertEqual(json_data["MY_ENV_VAR2"], "bar")
+        self.assertEqual(json_data["MY_ENV_VAR3"], "baz")
+
+    def install_conanfile_test(self):
+        """
+        Test the availability of user-defined generators from a conanfile
+        """
+        client = TestClient()
+
+        # Set up generator
+        test_generator_contents = textwrap.dedent("""
+            from conans import ConanFile
+            from conans.model import Generator
+
+            class user_defined(Generator):
+                @property
+                def filename(self):
+                    return "userdefined.txt"
+
+                @property
+                def content(self):
+                    return "user_defined contents"
+            """)
+        # Save generator to cache_folder/generators.
+        generator_path = os.path.join(client.cache.generators_path, "user_defined_generator.py")
+        save(generator_path, test_generator_contents)
+
+        conanfile_py = textwrap.dedent("""
+            from conans import ConanFile, tools
+
+            class HelloConan(ConanFile):
+                generators = "user_defined"
+
+                def package(self):
+                    self.output.info("create_conanfile_test: {}"
+                                      .format(tools.load("userdefined.txt")))
+                """)
+        client.save({"conanfile.py": conanfile_py})
+
+        # Test the install using a conanfile
+        client.run("install . --build")
+
+        data = client.load("userdefined.txt")
+        self.assertEqual(data, "user_defined contents")
+
+        # Test the install using a conanfile
+        client.run("create . Hello/0.1@lasote/testing")
+        self.assertIn("Generator user_defined created userdefined.txt", client.out)
+        self.assertIn("create_conanfile_test: user_defined contents", client.out)
+
+        client.run("install Hello/0.1@lasote/testing --build")
diff --git a/conans/test/unittests/client/cmd/ast_replacement_scm_test.py b/conans/test/unittests/client/cmd/ast_replacement_scm_test.py
--- a/conans/test/unittests/client/cmd/ast_replacement_scm_test.py
+++ b/conans/test/unittests/client/cmd/ast_replacement_scm_test.py
@@ -153,4 +153,3 @@ def test_multiline_comment_and_attribute(self):
         # FIXME: We lost the multiline comment
         self.assertIn("    url=23", load(conanfile))
         # self.assertIn(comment, load(conanfile))
-