Repository: conan-io/conan
Pull request: Feature: Generator for python environment
Changelog: Feature: Virtual environment generator for gathering only the PYTHONPATH.
Docs: https://github.com/conan-io/docs/pull/1369

This pull-request closes #5157

- [X] Refer to the issue that supports this Pull Request.
- [X] If the issue has missing info, explain the purpose/use case/pain/need that covers this Pull Request.
- [X] I've read the [Contributing guide](https://github.com/conan-io/conan/blob/develop/.github/CONTRIBUTING.md).
- [X] I've followed the PEP8 style guides for Python code.
- [X] I've opened another PR in the Conan docs repo to the ``develop`` branch, documenting this one. 

<sup>**Note:** By default this PR will skip the slower tests and will use a limited set of python versions. Check [here](https://github.com/conan-io/conan/blob/develop/.github/PR_INCREASE_TESTING.md) how to increase the testing level by writing some tags in the current PR body text.</sup>


Problem statement:
[suggestion] Add PYTHONPATH to virtualrunenv
Adding PYTHONPATH would make it sufficient to activate a virtualrunenv to use python modules in conan packages that also have dynamic library dependencies.

Without PYTHONPATH (as it is now) both a virtualrunenv and virtualenv needs to be activated for such python modules to work.

I would like to contribute to this feature.

Hints from issue discussion:
Not sure, but might be related to this issue: https://github.com/conan-io/conan/issues/5117
The generators `virtualenv` and `virtualrunenv` are separated because they manage different things. With #5158 you are mixing them. I agree it would be only one step but that is a design decision, and, in my opinion, is random, why PYTHONPATH and not others?
We might consider unifying both generators for Conan 2.0

I would like to understand better the use case, why aren't you using [python_requires](https://docs.conan.io/en/latest/extending/python_requires.html#python-requires)?
> The generators `virtualenv` and `virtualrunenv` are separated because they manage different things. With #5158 you are mixing them. I agree it would be only one step but that is a design decision, and, in my opinion, is random, why PYTHONPATH and not others?
> We might consider unifying both generators for Conan 2.0
> 
> I would like to understand better the use case, why aren't you using [python_requires](https://docs.conan.io/en/latest/extending/python_requires.html#python-requires)?

I'm creating python bindings with [pybind11](https://github.com/pybind/pybind11) which are used in different python applications.
The python bindings are distributed as Conan packages.
Can `python_requires` be used in generic python code (not `conanfile.py` files)?
`LD_LIBRARY_PATH` is also needed since the python bindings use some shared libraries.
Interesting, the python extensions management is a topic we would love to document and write a blog post about it. 
@atilag you might be interested.

I would love to unify the `virtualenv` generators, but in the meantime, should we consider to add another one for this use case? `virtualenv_python` or similar? @memsharded feedback needed here.


I have been thinking about unifying the virtualenv generators, and the thing is I still see a use case where users want the "runtime-only" information, without all the extra configuration that might be necessary for building.

While we have a look for it for Conan 2.0, yes, I wouldn't oppose to create a ``virtualenv_python``, if it is useful for this use case.
I see. It would be usefull to have a `virtualenv_python` in the meantime.

What is needed for creating a `virtualenv_python` generator?

Something like this?
 - Create `conans/client/generators/virtualenv_python.py` and add it in `conans/client/generators/__init__.py`
 - Create `docs/reference/generators/virtualenv_python.rst` and list it in `docs/reference/generators.rst`
Yes, that is the way to go 👍 
Just another suggestion...

Could we add a feature to specify a list of python modules which will be installed (using pip) in this python virtual environment?  We have packages which are dependent on several python modules which each have their own secondary python module dependencies.  We have been struggling for some time on how to solve this.  We don't want to install to the system in case other projects are using different versions of the modules.

Also, if the installed modules could somehow be added to the dependency graph, that would be even better.

Does this sound feasible?

Commit message: Merge pull request #1 from lasote/feature/python_virtualenv

Feature/python virtualenv

Patch to review:
diff --git a/conans/client/generators/__init__.py b/conans/client/generators/__init__.py
index dde846ad1e0..3fad34f07b5 100644
--- a/conans/client/generators/__init__.py
+++ b/conans/client/generators/__init__.py
@@ -24,6 +24,7 @@
 from .text import TXTGenerator
 from .virtualbuildenv import VirtualBuildEnvGenerator
 from .virtualenv import VirtualEnvGenerator
+from .virtualenv_python import VirtualEnvPythonGenerator
 from .virtualrunenv import VirtualRunEnvGenerator
 from .visualstudio import VisualStudioGenerator
 from .visualstudio_multi import VisualStudioMultiGenerator
@@ -70,6 +71,7 @@ def __getitem__(self, key):
 registered_generators.add("xcode", XCodeGenerator)
 registered_generators.add("ycm", YouCompleteMeGenerator)
 registered_generators.add("virtualenv", VirtualEnvGenerator)
+registered_generators.add("virtualenv_python", VirtualEnvPythonGenerator)
 registered_generators.add("virtualbuildenv", VirtualBuildEnvGenerator)
 registered_generators.add("virtualrunenv", VirtualRunEnvGenerator)
 registered_generators.add("boost-build", BoostBuildGenerator)
diff --git a/conans/client/generators/virtualenv_python.py b/conans/client/generators/virtualenv_python.py
new file mode 100644
index 00000000000..7e66cea2e21
--- /dev/null
+++ b/conans/client/generators/virtualenv_python.py
@@ -0,0 +1,22 @@
+from conans.client.generators.virtualrunenv import VirtualRunEnvGenerator
+
+
+class VirtualEnvPythonGenerator(VirtualRunEnvGenerator):
+
+    def __init__(self, conanfile):
+        super(VirtualEnvPythonGenerator, self).__init__(conanfile)
+        self.venv_name = "conanenvpython"
+        ppath = conanfile.env.get("PYTHONPATH")
+        if ppath:
+            self.env.update({"PYTHONPATH": [ppath, ] if not isinstance(ppath, list) else ppath})
+
+    @property
+    def content(self):
+        tmp = super(VirtualEnvPythonGenerator, self).content
+        ret = {}
+        for name, value in tmp.items():
+            tmp = name.split(".")
+            ret["%s_python.%s" % (tmp[0], tmp[1])] = value
+
+        return ret
+
diff --git a/conans/test/functional/generators/virtualenv_python_test.py b/conans/test/functional/generators/virtualenv_python_test.py
new file mode 100644
index 00000000000..7c7eacf6e95
--- /dev/null
+++ b/conans/test/functional/generators/virtualenv_python_test.py
@@ -0,0 +1,102 @@
+import platform
+import unittest
+import os
+
+from conans.util.files import load
+
+from conans.test.utils.tools import TestClient, GenConanfile
+
+
+class VirtualEnvPythonGeneratorTest(unittest.TestCase):
+
+    def simple_value_test(self):
+        client = TestClient()
+        dep1 = """
+import os
+from conans import ConanFile
+
+class BaseConan(ConanFile):
+    name = "base"
+    version = "0.1"
+
+    def package_info(self):
+        self.env_info.PYTHONPATH="/path/to/something"
+        self.env_info.LD_LIBRARY_PATH="/path/ld_library"
+        self.env_info.DYLD_LIBRARY_PATH="/path/dyld_library"
+        self.env_info.PATH="/path/path"
+        self.env_info.OTHER="23"
+"""
+
+        base = '''
+[requires]
+base/0.1
+[generators]
+virtualenv
+    '''
+        client.save({"conanfile.py": dep1})
+        client.run("create . ")
+        client.save({"conanfile.txt": base}, clean_first=True)
+        client.run("install . -g virtualenv_python")
+        name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_run_python.bat"
+        contents = load(os.path.join(client.current_folder, name))
+        self.assertNotIn("OTHER", contents)
+        self.assertIn("PATH=", contents)
+        self.assertIn("LD_LIBRARY_PATH=", contents)
+        self.assertIn("DYLD_LIBRARY_PATH=", contents)
+
+        if platform.system() != "Windows":
+
+            self.assertIn('PYTHONPATH="/path/to/something"${PYTHONPATH+:$PYTHONPATH}', contents)
+        else:
+            self.assertIn('SET PYTHONPATH=/path/to/something;%PYTHONPATH%', contents)
+
+    def multiple_value_test(self):
+            client = TestClient()
+            dep1 = """
+from conans import ConanFile
+
+class BaseConan(ConanFile):
+    name = "base"
+    version = "0.1"
+
+    def package_info(self):
+        self.env_info.PYTHONPATH=["/path/to/something", "/otherpath"]
+        self.env_info.OTHER="23"
+"""
+
+            base = '''
+    [requires]
+    base/0.1
+    [generators]
+    virtualenv
+        '''
+            client.save({"conanfile.py": dep1})
+            client.run("create . ")
+            client.save({"conanfile.txt": base}, clean_first=True)
+            client.run("install . -g virtualenv_python")
+            name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_run_python.bat"
+            contents = load(os.path.join(client.current_folder, name))
+            self.assertNotIn("OTHER", contents)
+            if platform.system() != "Windows":
+                self.assertIn('PYTHONPATH="/path/to/something":"/otherpath"'
+                              '${PYTHONPATH+:$PYTHONPATH}', contents)
+            else:
+                self.assertIn('SET PYTHONPATH=/path/to/something;/otherpath;%PYTHONPATH%', contents)
+
+    def no_value_declared_test(self):
+        client = TestClient()
+        dep1 = GenConanfile()
+
+        base = '''
+[requires]
+base/0.1
+[generators]
+virtualenv
+    '''
+        client.save({"conanfile.py": dep1})
+        client.run("create . base/0.1@")
+        client.save({"conanfile.txt": base}, clean_first=True)
+        client.run("install . -g virtualenv_python")
+        name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_python.bat"
+        contents = load(os.path.join(client.current_folder, name))
+        self.assertNotIn("PYTHONPATH", contents)
diff --git a/conans/test/unittests/client/generators/virtualenv_python_test.py b/conans/test/unittests/client/generators/virtualenv_python_test.py
new file mode 100644
index 00000000000..fa879754f5c
--- /dev/null
+++ b/conans/test/unittests/client/generators/virtualenv_python_test.py
@@ -0,0 +1,26 @@
+import unittest
+
+from conans import ConanFile, Settings
+from conans.client.generators.virtualenv_python import VirtualEnvPythonGenerator
+from conans.model.env_info import DepsEnvInfo
+from conans.model.env_info import EnvValues
+from conans.test.utils.tools import TestBufferConanOutput
+
+
+class VirtualEnvPythonGeneratorTest(unittest.TestCase):
+
+    def pythonpath_test(self):
+        """
+        Check PYTHONPATH env variable
+        """
+        conanfile = ConanFile(TestBufferConanOutput(), None)
+        conanfile.initialize(Settings({}), EnvValues.loads("PYTHONPATH=[1,2,three]"))
+        conanfile.deps_env_info = DepsEnvInfo.loads(
+            '[ENV_A]\nPYTHONPATH=["DepAPath"]\n[ENV_B]\nPYTHONPATH=["DepBPath"]'
+        )
+        gen = VirtualEnvPythonGenerator(conanfile)
+        content = gen.content
+
+        self.assertIn('PYTHONPATH="1":"2":"three":"DepAPath":"DepBPath"${PYTHONPATH+:$PYTHONPATH}',
+                      content["activate_run_python.sh"])
+


Code review comments:
diff_hunk: @@ -0,0 +1,102 @@
+import platform
+import unittest
+import os
+
+from conans.util.files import load
+
+from conans.test.utils.tools import TestClient, GenConanfile
+
+
+class VirtualEnvPythonGeneratorTest(unittest.TestCase):
+
+    def simple_value_test(self):
+        client = TestClient()
+        dep1 = """
+import os
+from conans import ConanFile
+
+class BaseConan(ConanFile):
+    name = "base"
+    version = "0.1"
+
+    def package_info(self):
+        self.env_info.PYTHONPATH="/path/to/something"
+        self.env_info.LD_LIBRARY_PATH="/path/ld_library"
+        self.env_info.DYLD_LIBRARY_PATH="/path/dyld_library"
+        self.env_info.PATH="/path/path"
+        self.env_info.OTHER="23"
+"""
+
+        base = '''
+[requires]
+base/0.1
+[generators]
+virtualenv
+    '''
+        client.save({"conanfile.py": dep1})
+        client.run("create . ")
+        client.save({"conanfile.txt": base}, clean_first=True)
+        client.run("install . -g virtualenv_python")
+        name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_run_python.bat"
+        contents = load(os.path.join(client.current_folder, name))
+        self.assertNotIn("OTHER", contents)
+        self.assertIn("PATH=", contents)
+        self.assertIn("LD_LIBRARY_PATH=", contents)
+        self.assertIn("DYLD_LIBRARY_PATH=", contents)
+
+        if platform.system() != "Windows":
+
+            self.assertIn('PYTHONPATH="/path/to/something"${PYTHONPATH+:$PYTHONPATH}', contents)
+        else:
+            self.assertIn('SET PYTHONPATH=/path/to/something;%PYTHONPATH%', contents)
+
+    def multiple_value_test(self):
+            client = TestClient()
+            dep1 = """
+from conans import ConanFile
+
+class BaseConan(ConanFile):
+    name = "base"
+    version = "0.1"
+
+    def package_info(self):
+        self.env_info.PYTHONPATH=["/path/to/something", "/otherpath"]
+        self.env_info.OTHER="23"
+"""
+
+            base = '''
+    [requires]
+    base/0.1
+    [generators]
+    virtualenv
+        '''
+            client.save({"conanfile.py": dep1})
+            client.run("create . ")
+            client.save({"conanfile.txt": base}, clean_first=True)
+            client.run("install . -g virtualenv_python")
+            name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_run_python.bat"
+            contents = load(os.path.join(client.current_folder, name))
+            self.assertNotIn("OTHER", contents)
+            if platform.system() != "Windows":
+                self.assertIn('PYTHONPATH="/path/to/something":"/otherpath"'
+                              '${PYTHONPATH+:$PYTHONPATH}', contents)
+            else:
+                self.assertIn('SET PYTHONPATH=/path/to/something;/otherpath;%PYTHONPATH%', contents)
+
+    def no_value_declared_test(self):
+        client = TestClient()
+        dep1 = GenConanfile()
+
+        base = '''
+[requires]
+base/0.1
+[generators]
+virtualenv
+    '''
+        client.save({"conanfile.py": dep1})
+        client.run("create . base/0.1@")
+        client.save({"conanfile.txt": base}, clean_first=True)
+        client.run("install . -g virtualenv_python")
+        name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_python.bat"
path: conans/test/functional/generators/virtualenv_python_test.py
text: @user1:
```suggestion
        name = "activate_run_python.sh" if platform.system() != "Windows" else "activate_run_python.bat"
```

@user1:
Please @author accept this commit. I forgot to change a file name.