Repository: conan-io/conan

Issue:
[bug] conanfile loader "crashes" because of missing __file__ attribute in Python 3.7(+)
Hi there,
I am using multiple Python versions (e.g. 3.6, 3.7). My primary Python version is 3.6.
But occationally I execute Python version 3.7. With 3.6 there where no problems. 3.7 does not work.

I have 2 projects. One (```pkg_a```) gets ```python_requires```d into the other (```pkg_b```).
pkg_b inherits from a python file that gets exported with pkg_a. Now Python 3.6 everything works fine. In Python 3.7 I get an exception.

Crawling through the conan code I identified (at least I think) the code where the solution lies.
At [this code line](https://github.com/conan-io/conan/blob/4fdebfb79ce12f956b4be512253e8479e9d6bb17/conans/client/loader.py#L265) conan can catch an exception if the python module has no ```__file__``` attribute.
But in Python 3.7 I get a ```TypeError``` exception instead of an ```AttributeError``` exception.

The [Python docs describe](https://docs.python.org/3.6/library/exceptions.html#AttributeError) ([see](https://docs.python.org/3.6/library/exceptions.html#AttributeError)) that there could be raised a ```TypeError``` exception.

In turn I modified the code line [265](https://github.com/conan-io/conan/blob/4fdebfb79ce12f956b4be512253e8479e9d6bb17/conans/client/loader.py#L265) in the conan [loader.py](https://github.com/conan-io/conan/blob/develop/conans/client/loader.py).

From:
```python
except AttributeError: # some module doesn't have __file__
    pass
```

To:
```python
except AttributeError: # some module doesn't have __file__
    pass
except TypeError: # Python 3.7: some module doesn't have __file__
    pass
```

And everything worked like before.
I do not know if this is a Python or Conan problem. But at the moment this is a real issue as noone with a version of Python 3.7+ can execute my recipe.

For reference I made a minimal example to track down the issue.
[py36_py37_exception.zip](https://github.com/conan-io/conan/files/2915266/py36_py37_exception.zip)

Add tests and simplify code for parse_conanfile
Changelog: (Feature | Fix | Bugfix): Describe here your pull request
Docs: https://github.com/conan-io/docs/pull/XXXX

closes #4636 

 * Tests: add more use cases for `conanfiles` with adyacent files.
 * Changes in code: just a proposal (to be tested in all versions)

@PYVERS: Macos@py27, Windows@py36, Linux@py27, py34



Discussion:
Hi! Thanks! It looks like a bug, indeed. Just for clarification, the error is not the `module.__file__` raising an `AttributeError` (it returns `None`), but the `os.path.dirname` raising a `TypeError` for the invalid input argument.

I'm writing a test to reproduce this behavior

I have had a look, there are a couple of issues removing this code:

- Every import of every recipe that is not previously used in Conan, will be parsed and imported everytime it is used. If a dependency graph of 100 packages uses an ``import myownlib``, the ``myownlib`` will be parsed and imported 100 times, which is a very expensive operation.
- Besides the performance, if for some reason someone is using a library that has global state or sync (lets say, a logger), it will fail, as every importer will have its own instance.

I will try to provide a test that proves this behavior.
OMG! @memsharded 🎉 🎉 
This is a failing test (passes in develop) for this PR:

```python
    def test_helpers_python_library(self):
        mylogger = """
f = open("mylogfile.txt", "w")
def save(data):
    f.write(data)
"""
        temp = temp_folder()
        save(os.path.join(temp, "mylogger.py"), mylogger)
        save(os.path.join(temp, "__init__.py"), "")
        conanfile = '''from conans import ConanFile
import mylogger
import os
class ConanFileToolsTest(ConanFile):
    %s
    def configure(self):
        mylogger.save("source %%s!!!" %% self.name)
'''

        client = TestClient()
        try:
            sys.path.append(temp)

            client.save({"conanfile.py": conanfile % ""})
            client.run("export . liba/0.1@user/testing")
            client.save({"conanfile.py": conanfile % "requires = 'liba/0.1@user/testing'"})
            client.run("export . libb/0.1@user/testing")
            client.save({"conanfile.py": conanfile % "requires = 'libb/0.1@user/testing'"})

            client.run("info .")
            logged = load(os.path.join(client.current_folder, "mylogfile.txt"))
            self.assertIn("source libb!!!", logged)
            self.assertIn("source liba!!!", logged)
        finally:
            sys.path.remove(temp)
```
I'm not sure if it is expected behavior or not, if it is right the previous one (develop) or this one, but I like this one more. Let me try to explain myself.

The important thing here is to distinguish between two different scenarios:
 1. if the module has been previously loaded by Conan (like `requests`, `os`,...) then we want these `conanfiles.py` to use the module already imported. For example, in the line `from conans import ConanFile` we want the recipe to use the `ConanFile` from the _global python libraries_, not from a module/file next to the `conanfile.py` we are loading that happens to be called `conans.py`.
 2. but, if the recipe is introducing a new module, never seen by python before, then I would expect it to not interfere with modules/files with the same name that are within other recipes.

----

## Scenario 2: the recipes are introducing the module `mylogger` to the python interpreter

So, let me simplify your test a little bit, removing all the machinery from Conan:

```python
    def test_global_import(self):
        # It will be a mylogger.py
        mylogger = textwrap.dedent("""
            import os
            
            def get_filename():
                return os.path.abspath("mylogfile.txt")
        
            filename = get_filename()
            print("Exists '{}': {}".format(filename, os.path.exists(filename)))
            f = open(filename, "a")

            def save(data):
                f.write(data)
            """)

        # It will be like the `conanfile.py`
        content = textwrap.dedent("""
            print("pre import - {value}")
            import mylogger
            print("post import - {value}")
            
            mylogger.save("{value}")
        """)

        tmp = temp_folder()
        sys.path.insert(0, tmp)
        try:
            with chdir(tmp):
                save("mylogger.py", mylogger)
                save("__init__py", "")
 
            # I will talk about these two lines afterwards, this is the 
            #    difference between the two scenarios presented above
            #print("\n>>>> Global import")
            #import mylogger
            #print(mylogger.get_filename())
            #print(mylogger.filename)

            def save_and_load(value):
                tmp = temp_folder()
                save(os.path.join(tmp, 'content.py'), content.format(value=value))
                loaded, module_id = _parse_conanfile(os.path.join(tmp, "content.py"))
                return loaded, module_id

            print("\n>>>> Running first: 42")
            loaded1, module_id1 = save_and_load(42)
            print(loaded1.mylogger.filename)
            print(loaded1.mylogger.get_filename())

            print("\n>>>> Running second: 23")
            loaded2, module_id2 = save_and_load(23)
            print(loaded1.mylogger.filename)
            print(loaded2.mylogger.get_filename())

            self.fail("AAA")
        finally:
            sys.path.pop(0)
```

The output for this test (in this branch) is the following:

```
>>>> Running first: 42
pre import - 42
Exists '/private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmpsz4xgmn5conans/path with spaces/mylogfile.txt': False
post import - 42
/private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmpsz4xgmn5conans/path with spaces/mylogfile.txt
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt

>>>> Running second: 23
pre import - 23
Exists '/private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmpzabgm7j7conans/path with spaces/mylogfile.txt': False
post import - 23
/private/var/folders/yq/14hmvxm96xd7gfgl37_tnrbh0000gn/T/tmpsz4xgmn5conans/path with spaces/mylogfile.txt
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt
```

The interpreter has never seen the `mylogger` module, so **it is adding it for each of the _conanfiles_** (the line `Exists ....` between the `pre import` and `post import` messages), each one will use a different path for the file (the temp-folder where each one has been created).

And I want it to add the module for each of the _conanfiles_ otherwise the path used for the `f` fileobject will be different depending on which of the _conanfiles_ is loaded first.

----


## Scenario 2: the global interpreter already knows about the module

Second escenario, just uncomment these lines:

```python
            print("\n>>>> Global import")
            import mylogger
            print(mylogger.get_filename())
            print(mylogger.filename)
```

Now the output is different:

```python
>>>> Global import
Exists '/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt': True
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt

>>>> Running first: 42
pre import - 42
post import - 42
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt

>>>> Running second: 23
pre import - 23
post import - 23
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt
/Users/jgsogo/dev/conan/conan/conans/test/unittests/client/loader/mylogfile.txt
```

Now the _python interpreter_ already knows about the `mylogger` module so it is not importing it again and all the _conanfiles_ are using the same file which has been computed in the first import (the global/Conan one).

Main points here:
 * **Isolation**: each recipe is isolated: if one of them imports something it won't affect other recipes.
 * **Performace**: if Conan imports something, it will only be imported once for all recipes.
> * **Isolation**: each recipe is isolated: if one of them imports something it won't affect other recipes.

That is the point. When you are using a global logging facility library, that you want all your recipes to be logging things (like build output), you cannot have an isolated, separated instance of the logger, because it will not achieve what you want. That is the reason such facility is installed as a system python library. It is not in your recipe, not in conan. And then the final result is different, if you are running from your IDE, that often does pre-load things in ``sys.modules`` you have one behavior, but if you run from the command line, you get a different one.

> * **Performace**: if Conan imports something, it will only be imported once for all recipes.

Yes, indeed that is my other concern. Parsing python is one of the slowest things in the whole conan processing. Having a large dependency graph (100, 200 packages), need to parse the same system libraries 100 or 200 times, could be insane.

It might not be a very common use case, but for sure we know users that are using such system (both system, and their own python utilities, installed at the system level) to extend Conan recipes.
> It might not be a very common use case, but for sure we know users that are using such system (both system, and their own python utilities, installed at the system level) to extend Conan recipes.

**This is the point** and this should be done through `python_requires`, otherwise, how can you be sure that the logging facility you are using is the one you wanted to use? The other option is to export this helpers/functions/utils with the `conanfile.exports` member in every recipe.

> if you are running from your IDE, that often does pre-load things in `sys.modules` you have one behavior, but if you run from the command line, you get a different one.

Here I agree with you too, but the only solution to this would be to isolate even more the loading of the `conanfiles`, before loading a `conanfile` we should remove all modules except the Conan requirements and Conan itself, and after loading the recipe restore it again... that's the reason we are providing `tools` curryficated. <-- I'm not proposing to do it right now, but we should consider it for full reproducibility.


> **This is the point** and this should be done through `python_requires`, otherwise, how can you be sure that the logging facility you are using is the one you wanted to use? The other option is to export this helpers/functions/utils with the `conanfile.exports` member in every recipe.

I don't think we can make users that decided to use third party python libraries, installed with pip, in their conan recipes to convert them to Conan python_requires.
Sure, we cannot, that's the reason I'm not proposing to do it now.

Nevertheless, I think that if a recipe contains a custom implementation, we should try to respect it; otherwise, the resulting behavior will depend on the import order of the different `conanfile`s or a new requirement in my graph can modify the behavior of some `conanfile` that are being loaded after, or we can get errors not shown before... and this is weird.

This test suite may explain it better than me (comments below):

```python
# coding=utf-8

import os
import sys
import textwrap
import unittest

from parameterized import parameterized

from conans.client.loader import _parse_conanfile
from conans.client.tools.files import save, chdir
from conans.test.utils.test_files import temp_folder


class ImportModuleLoaderTest(unittest.TestCase):

    def setUp(self):
        sys.modules.pop('fractions', None)
        self.assertFalse("fractions" in sys.modules.keys())

    @staticmethod
    def _create_module1():
        """ Module 1 provides its own implementation of 'fractions.Fraction' (returns float) """
        fractions = textwrap.dedent("""
            class Fraction:
               def __init__(self, num, den):
                   self._num = num
                   self._den = den
                   
               def __str__(self):
                   return "{}".format(self._num/float(self._den))
                
        """)

        conanfile = textwrap.dedent("""
            from fractions import Fraction
            
            def get_fraction_str():
                return str(Fraction(1, 2))  # returns "0.5"
        """)

        tmp = temp_folder()
        with chdir(tmp):
            save("conanfile.py", conanfile)
            save("fractions.py", fractions)
            loaded, module_id = _parse_conanfile(os.path.join(tmp, "conanfile.py"))
        return loaded, module_id, "0.5"

    @staticmethod
    def _create_module2():
        """ Module 2 uses the built-in implementation (returns the fraction itself) """
        conanfile = textwrap.dedent("""
                    from fractions import Fraction

                    def get_fraction_str():
                        return str(Fraction(1, 2))  # returns "1/2"
                """)

        tmp = temp_folder()
        with chdir(tmp):
            save("conanfile.py", conanfile)
            loaded, module_id = _parse_conanfile(os.path.join(tmp, "conanfile.py"))
        return loaded, module_id, "1/2"

    def test_module1(self):
        """ Module 1 has its own implementation of the `fractions.Fraction` class """
        loaded, module_id, result = self._create_module1()
        self.assertEqual(loaded.get_fraction_str(), result)

    def test_module2(self):
        """ Module 2 uses the built-in 'fractions' module """
        loaded, module_id, result = self._create_module2()
        self.assertEqual(loaded.get_fraction_str(), result)

    @parameterized.expand([(True, ), (False, )])
    def test_parse_order(self, inverse_load_order):
        """ The parse order of the conanfiles cannot affect the expected results """
        first, last = self._create_module1, self._create_module2
        if inverse_load_order:
            first, last = last, first

        loaded_first, _, result_first = first()
        loaded_last, _, result_last = last()

        self.assertNotEqual(result_first, result_last)
        self.assertEqual(loaded_first.get_fraction_str(), result_first)
        self.assertEqual(loaded_last.get_fraction_str(), result_last)

    def test_fail_for_custom(self):
        """ Module 1 has a custom implementation, but Conan already knows about the module """
        # Make module 'fractions' known to Conan itself.
        import fractions  # pylint: disable=unused-import
        self.assertTrue("fractions" in sys.modules.keys())

        loaded, module_id, result = self._create_module1()
        with self.assertRaisesRegexp(AssertionError, "'1/2' != '0.5'"):
            self.assertEqual(loaded.get_fraction_str(), result)
```

A brief comment about the tests:
 * `test_module1`: will work as long as Conan itself doesn't know about the module `fractions`. This problem also exists in the previous implementation.
 * `test_module2`: will always work
 * `test_parse_order`: **it works now**; it should fail in the previous implementation because there will be only one `fractions` module, so first loaded wins.
 * `test_fail_for_custom`: it _fails_ now exactly as it was failing before, the `fractions` module is know to Conan and we do not allow overriding it.

I think that with this change we get an improvement, more consistent behavior.
If you don't mind, I would like to deprecate/close this PR in favor of https://github.com/conan-io/conan/pull/4664, in that one all the tests are running and the different approaches are isolated in its own commit to show the results in the CI.

Resolving patch:
diff --git a/conans/client/hook_manager.py b/conans/client/hook_manager.py
--- a/conans/client/hook_manager.py
+++ b/conans/client/hook_manager.py
@@ -101,8 +101,14 @@ def _load_module_from_file(hook_path):
                 module = sys.modules[added]
                 if module:
                     try:
-                        folder = os.path.dirname(module.__file__)
-                    except AttributeError:  # some module doesn't have __file__
+                        try:
+                            # Most modules will have __file__ != None
+                            folder = os.path.dirname(module.__file__)
+                        except (AttributeError, TypeError):
+                            # But __file__ might not exist or equal None
+                            # Like some builtins and Namespace packages py3
+                            folder = module.__path__._path[0]
+                    except AttributeError:  # In case the module.__path__ doesn't exist
                         pass
                     else:
                         if folder.startswith(current_dir):
diff --git a/conans/client/loader.py b/conans/client/loader.py
--- a/conans/client/loader.py
+++ b/conans/client/loader.py
@@ -261,8 +261,14 @@ def _parse_conanfile(conan_file_path):
             module = sys.modules[added]
             if module:
                 try:
-                    folder = os.path.dirname(module.__file__)
-                except AttributeError:  # some module doesn't have __file__
+                    try:
+                        # Most modules will have __file__ != None
+                        folder = os.path.dirname(module.__file__)
+                    except (AttributeError, TypeError):
+                        # But __file__ might not exist or equal None
+                        # Like some builtins and Namespace packages py3
+                        folder = module.__path__._path[0]
+                except AttributeError:  # In case the module.__path__ doesn't exist
                     pass
                 else:
                     if folder.startswith(current_dir):


Test patch:
diff --git a/conans/test/unittests/client/conanfile_loader_test.py b/conans/test/unittests/client/conanfile_loader_test.py
--- a/conans/test/unittests/client/conanfile_loader_test.py
+++ b/conans/test/unittests/client/conanfile_loader_test.py
@@ -1,21 +1,27 @@
 import os
+import sys
+import textwrap
 import unittest
 from collections import OrderedDict
 
+import six
 from mock import Mock
 from mock.mock import call
+from parameterized import parameterized
 
 from conans.client.graph.python_requires import ConanPythonRequire
-from conans.client.loader import ConanFileLoader, ConanFileTextLoader
+from conans.client.loader import ConanFileLoader, ConanFileTextLoader,\
+    _parse_conanfile
+from conans.client.tools.files import chdir
 from conans.errors import ConanException
 from conans.model.options import OptionsValues
 from conans.model.profile import Profile
 from conans.model.requires import Requirements
 from conans.model.settings import Settings
 from conans.test.utils.test_files import temp_folder
-from conans.util.files import save
 from conans.test.utils.tools import test_processed_profile,\
     TestBufferConanOutput
+from conans.util.files import save
 
 
 class ConanLoaderTest(unittest.TestCase):
@@ -225,3 +231,102 @@ class MyTest(ConanFile):
         recipe = loader.load_consumer(conanfile_path,
                                       test_processed_profile(profile))
         self.assertIsNone(recipe.settings.os.value)
+
+
+class ImportModuleLoaderTest(unittest.TestCase):
+    @staticmethod
+    def _create_and_load(myfunc, value, subdir_name, add_subdir_init):
+        subdir_content = textwrap.dedent("""
+            def get_value():
+                return {value}
+            def {myfunc}():
+                return "{myfunc}"
+        """)
+
+        side_content = textwrap.dedent("""
+            def get_side_value():
+                return {value}
+            def side_{myfunc}():
+                return "{myfunc}"
+        """)
+
+        conanfile = textwrap.dedent("""
+            import pickle
+            from {subdir}.api import get_value, {myfunc}
+            from file import get_side_value, side_{myfunc}
+            from fractions import Fraction
+            def conanfile_func():
+                return get_value(), {myfunc}(), get_side_value(), side_{myfunc}(), str(Fraction(1,1))
+        """)
+        expected_return = (value, myfunc, value, myfunc, "1")
+
+        tmp = temp_folder()
+        with chdir(tmp):
+            save("conanfile.py", conanfile.format(value=value, myfunc=myfunc, subdir=subdir_name))
+            save("file.py", side_content.format(value=value, myfunc=myfunc))
+            save("{}/api.py".format(subdir_name), subdir_content.format(value=value, myfunc=myfunc))
+            if add_subdir_init:
+                save("__init__.py", "")
+                save("{}/__init__.py".format(subdir_name), "")
+
+        loaded, module_id = _parse_conanfile(os.path.join(tmp, "conanfile.py"))
+        return loaded, module_id, expected_return
+
+    @parameterized.expand([(True, False), (False, True), (False, False)])
+    @unittest.skipIf(six.PY2, "Python 2 requires __init__.py file in modules")
+    def test_py3_recipe_colliding_init_filenames(self, sub1, sub2):
+        myfunc1, value1 = "recipe1", 42
+        myfunc2, value2 = "recipe2", 23
+        loaded1, module_id1, exp_ret1 = self._create_and_load(myfunc1, value1, "subdir", sub1)
+        loaded2, module_id2, exp_ret2 = self._create_and_load(myfunc2, value2, "subdir", sub2)
+
+        self.assertNotEqual(module_id1, module_id2)
+        self.assertEqual(loaded1.conanfile_func(), exp_ret1)
+        self.assertEqual(loaded2.conanfile_func(), exp_ret2)
+
+    def test_recipe_colliding_filenames(self):
+        myfunc1, value1 = "recipe1", 42
+        myfunc2, value2 = "recipe2", 23
+        loaded1, module_id1, exp_ret1 = self._create_and_load(myfunc1, value1, "subdir", True)
+        loaded2, module_id2, exp_ret2 = self._create_and_load(myfunc2, value2, "subdir", True)
+
+        self.assertNotEqual(module_id1, module_id2)
+        self.assertEqual(loaded1.conanfile_func(), exp_ret1)
+        self.assertEqual(loaded2.conanfile_func(), exp_ret2)
+
+    @parameterized.expand([(True, ), (False, )])
+    def test_wrong_imports(self, add_subdir_init):
+        myfunc1, value1 = "recipe1", 42
+
+        # Item imported does not exist, but file exists
+        with self.assertRaisesRegexp(ConanException, "Unable to load conanfile in"):
+            self._create_and_load(myfunc1, value1, "requests", add_subdir_init)
+
+        # File does not exists in already existing module
+        with self.assertRaisesRegexp(ConanException, "Unable to load conanfile in"):
+            self._create_and_load(myfunc1, value1, "conans", add_subdir_init)
+
+    def test_helpers_python_library(self):
+        mylogger = """
+value = ""
+def append(data):
+    global value
+    value += data
+"""
+        temp = temp_folder()
+        save(os.path.join(temp, "myconanlogger.py"), mylogger)
+
+        conanfile = "import myconanlogger"
+        temp1 = temp_folder()
+        save(os.path.join(temp1, "conanfile.py"), conanfile)
+        temp2 = temp_folder()
+        save(os.path.join(temp2, "conanfile.py"), conanfile)
+
+        try:
+            sys.path.append(temp)
+            loaded1, _ = _parse_conanfile(os.path.join(temp1, "conanfile.py"))
+            loaded2, _ = _parse_conanfile(os.path.join(temp2, "conanfile.py"))
+            self.assertIs(loaded1.myconanlogger, loaded2.myconanlogger)
+            self.assertIs(loaded1.myconanlogger.value, loaded2.myconanlogger.value)
+        finally:
+            sys.path.remove(temp)