Repository: easybuilders/easybuild-framework

Issue:
Issue running tests
```
[mboisson@build-node easybuild-framework]$ python -O -m test.framework.suite test_update
INFO: This is (based on) vsc.install.shared_setup 0.10.17
Traceback (most recent call last):
  File "/cvmfs/soft.computecanada.ca/nix/store/4x0hqnpd0hfh62m3apkxmz8hz3hlsikx-python-2.7.13-env/lib/python2.7/runpy.py", line 174, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/cvmfs/soft.computecanada.ca/nix/store/4x0hqnpd0hfh62m3apkxmz8hz3hlsikx-python-2.7.13-env/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/home/mboisson/git/easybuild-framework/test/framework/suite.py", line 44, in <module>
    import test.framework.asyncprocess as a
  File "/home/mboisson/git/easybuild-framework/test/framework/asyncprocess.py", line 34, in <module>
    from test.framework.utilities import EnhancedTestCase
  File "/home/mboisson/git/easybuild-framework/test/framework/utilities.py", line 71, in <module>
    go = EasyBuildOptions(go_useconfigfiles=False)
  File "easybuild/tools/options.py", line 246, in __init__
    super(EasyBuildOptions, self).__init__(*args, **kwargs)
  File "/cvmfs/soft.computecanada.ca/nix/store/4x0hqnpd0hfh62m3apkxmz8hz3hlsikx-python-2.7.13-env/lib/python2.7/site-packages/vsc/utils/generaloption.py", line 982, in __init__
    self.main_options()
  File "/cvmfs/soft.computecanada.ca/nix/store/4x0hqnpd0hfh62m3apkxmz8hz3hlsikx-python-2.7.13-env/lib/python2.7/site-packages/vsc/utils/generaloption.py", line 1058, in main_options
    fn()
  File "easybuild/tools/options.py", line 493, in config_options
    'choice', 'store', DEFAULT_REPOSITORY, sorted(avail_repositories().keys())),
  File "easybuild/tools/repository/repository.py", line 152, in avail_repositories
    raise EasyBuildError("avail_repositories: FileRepository missing from list of repositories")
easybuild.tools.build_log.EasyBuildError: 'avail_repositories: FileRepository missing from list of repositories'
``` 

Opening an issue for @boegel 


Discussion:
Weirdly I do not see the same issue on the same node. I have run various test cases there many times.
Maybe it's just a config in my account then. 
seems like `easybuild_framework.egg-info` needs to be there, but is not. 
So basically for some reason I don't fully understand (I probably did something related to setuptools in January) my git checkout has a `easybuild_framework.egg-info` directory but Maxime has not. Once we copied it across, the tests worked.
How was this `easybuild-framework` installed done exactly?
`git clone git@github.com:ComputeCanada/easybuild-framework.git` 
(and added easybuilders as a second remote) 
@mboisson There must be more to it... What about `vsc-base`? How is that picked up? `$PYTHONPATH`?
I think we have vsc-base installed in Nix. 
@victorusu was running into the same error message. In his case it happened because i) he was running the tests from the `easybuild-framework` directory, ii) he had a `setuptools.pth` available in `$PYTHONPATH` which seems to have as a side-effect that the current working directory is not included in the Python search path (`sys.path`).
See #2742 for more details on the specific problem that @victorusu was hitting, could be a duplicate of this problem?

Resolving patch:
diff --git a/easybuild/tools/utilities.py b/easybuild/tools/utilities.py
--- a/easybuild/tools/utilities.py
+++ b/easybuild/tools/utilities.py
@@ -111,7 +111,16 @@ def import_available_modules(namespace):
     """
     modules = []
     for path in sys.path:
-        for module in sorted(glob.glob(os.path.sep.join([path] + namespace.split('.') + ['*.py']))):
+
+        cand_modpath_glob = os.path.sep.join([path] + namespace.split('.') + ['*.py'])
+
+        # if sys.path entry being considered is the empty string
+        # (which corresponds to Python packages/modules in current working directory being considered),
+        # we need to strip off / from the start of the path
+        if path == '' and cand_modpath_glob.startswith(os.path.sep):
+            cand_modpath_glob = cand_modpath_glob.lstrip(os.path.sep)
+
+        for module in sorted(glob.glob(cand_modpath_glob)):
             if not module.endswith('__init__.py'):
                 mod_name = module.split(os.path.sep)[-1].split('.')[0]
                 modpath = '.'.join([namespace, mod_name])
@@ -120,7 +129,10 @@ def import_available_modules(namespace):
                     mod = __import__(modpath, globals(), locals(), [''])
                 except ImportError as err:
                     raise EasyBuildError("import_available_modules: Failed to import %s: %s", modpath, err)
-                modules.append(mod)
+
+                if mod not in modules:
+                    modules.append(mod)
+
     return modules
 
 


Test patch:
diff --git a/test/framework/general.py b/test/framework/general.py
--- a/test/framework/general.py
+++ b/test/framework/general.py
@@ -36,9 +36,10 @@
 import vsc.utils.generaloption
 
 import easybuild.framework
+import easybuild.tools.repository.filerepo
 from easybuild.tools.build_log import EasyBuildError
-from easybuild.tools.filetools import read_file
-from easybuild.tools.utilities import only_if_module_is_available
+from easybuild.tools.filetools import change_dir, mkdir, read_file, write_file
+from easybuild.tools.utilities import import_available_modules, only_if_module_is_available
 
 
 class GeneralTest(EnhancedTestCase):
@@ -132,6 +133,32 @@ def test_docstrings(self):
                 for regex in docstring_regexes:
                     self.assertFalse(regex.search(txt), "No match for '%s' in %s" % (regex.pattern, path))
 
+    def test_import_available_modules(self):
+        """Test for import_available_modules function."""
+
+        res = import_available_modules('easybuild.tools.repository')
+        self.assertEqual(len(res), 5)
+        # don't check all, since some required specific Python packages to be installed...
+        self.assertTrue(easybuild.tools.repository.filerepo in res)
+
+        # replicate situation where import_available_modules failed when running in directory where modules are located
+        # cfr. https://github.com/easybuilders/easybuild-framework/issues/2659
+        #      and https://github.com/easybuilders/easybuild-framework/issues/2742
+        test123 = os.path.join(self.test_prefix, 'test123')
+        mkdir(test123)
+        write_file(os.path.join(test123, '__init__.py'), '')
+        write_file(os.path.join(test123, 'one.py'), '')
+        write_file(os.path.join(test123, 'two.py'), '')
+        write_file(os.path.join(test123, 'three.py'), '')
+
+        change_dir(self.test_prefix)
+        res = import_available_modules('test123')
+
+        import test123.one
+        import test123.two
+        import test123.three
+        self.assertEqual([test123.one, test123.three, test123.two], res)
+
 
 def suite():
     """ returns all the testcases in this module """