Repository: easybuilders/easybuild-framework

Issue:
Pushing easyconfigs to local git repo fails in 4.5.0
After we upgraded our EasyBuild installation to `4.5.0` we have noticed that the feature we use to send the easyconfigs and build logs to an internal git repo is no longer working. We have an ssh-key for our build user that works if we do a build with a previous EasyBuild version (`4.4.2` for example) but no longer works on `4.5.0`. At the end of the build we get:
```
/scicore/home/scicore/easybuil@bc2-gl.bc2.unibas.ch's password:
```
There seems to be some change in the way the `repositorypath` configuration option is processed, which in our config is:
```
repositorypath=git@bc2-gl.bc2.unibas.ch:easybuild/installed_easyconfigs_tests.git
```


@pescobar mentioned that this commit might be related https://github.com/easybuilders/easybuild-framework/commit/3252979f29b1b92a7223f5163cef02c7c03fa564


Discussion:
@i-mtz It looks like we're not taking into account that the `repositorypath` value may be something totally different than a simple path value...

Can you share the output of `eb --show-config` with both EasyBuild 4.4.2 and 4.5.0?
@i-mtz @pescobar The changes made in #3893 should fix the problem you're seeing, can you confirm?
@boegel, yes,  #3893 fixes the issue for us.
Thanks!
Another user mentioned issues and was using
`ssh://git@x.x.x.edu:2222/xxx/easybuild/slurm-cluster/deployed-recipes.git`
And, surely, some might be using local git repos with other usernames. I don't think `path.startswith('git@')` is good enough here.
One option would be to check with regexp if we have to modify that function without https://stackoverflow.com/questions/2514859/regular-expression-for-git-repository seems very robust.
How about just checking with `[a-z][@:]` for URL-like paths? That'll probably work just fine in practice?
Sure
Though, technically, can't you have a .ssh/config that specifies the username for you? So it's not a guarantee.
An alternative could perhaps be to check if the value corresponding with an existing absolute path, and if it doesn't, then don't try to "cast" it to an absolute path at all?
Isn't the problem the code in _postprocess_config (tools/options.py) where is does
```
        repositorypath = self.options.repositorypath
        if isinstance(repositorypath, (list, tuple)) and len(repositorypath) == 2:
            abs_path = self.get_cfg_opt_abs_path('repositorypath', repositorypath[0])
            self.options.repositorypath = (abs_path, repositorypath[1])
        else:
            path_opt_names.append('repositorypath')
```
that it should only do that if self.options.repository is FileRepository ? And absolutely not if it is GitRepository ?
> that it should only do that if self.options.repository is FileRepository ? And absolutely not if it is GitRepository ?

No, because `GitRepository` can also be local (so specified as a path), doesn't have to be remote (specified as a URL of some kind)
Another way out could be to simply skip the whole make-sure-it-is-an-absolute-path business for `repositorypath`, since it's quite unlikely that a relative path is specified there anyway?

The changes made in #3832 were mainly to prevent silly mistakes like in #3619, but for `repositorypath` it's probably much less likely that it's a problem...

Resolving patch:
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -1092,22 +1092,20 @@ def _ensure_abs_path(self, opt_name):
     def _postprocess_config(self):
         """Postprocessing of configuration options"""
 
-        # resolve relative paths for configuration options that specify a location;
+        # resolve relative paths for configuration options that specify a location,
+        # to avoid incorrect paths being used when EasyBuild changes the current working directory
+        # (see https://github.com/easybuilders/easybuild-framework/issues/3619);
         # ensuring absolute paths for 'robot' is handled separately below,
-        # because we need to be careful with the argument pass to --robot
+        # because we need to be careful with the argument pass to --robot;
+        # note: repositorypath is purposely not listed here, because it's a special case:
+        # - the value could consist of a 2-tuple (<path>, <relative_subdir>);
+        # - the <path> could also specify the location of a *remote* (Git( repository,
+        #   which can be done in variety of formats (git@<url>:<org>/<repo>), https://<url>, etc.)
+        #   (see also https://github.com/easybuilders/easybuild-framework/issues/3892);
         path_opt_names = ['buildpath', 'containerpath', 'git_working_dirs_path', 'installpath',
                           'installpath_modules', 'installpath_software', 'prefix', 'packagepath',
                           'robot_paths', 'sourcepath']
 
-        # repositorypath is a special case: only first part is a path;
-        # 2nd (optional) part is a relative subdir and should not be resolved to an absolute path!
-        repositorypath = self.options.repositorypath
-        if isinstance(repositorypath, (list, tuple)) and len(repositorypath) == 2:
-            abs_path = self.get_cfg_opt_abs_path('repositorypath', repositorypath[0])
-            self.options.repositorypath = (abs_path, repositorypath[1])
-        else:
-            path_opt_names.append('repositorypath')
-
         for opt_name in path_opt_names:
             self._ensure_abs_path(opt_name)
 


Test patch:
diff --git a/test/framework/config.py b/test/framework/config.py
--- a/test/framework/config.py
+++ b/test/framework/config.py
@@ -302,7 +302,7 @@ def test_generaloption_config_file(self):
         self.assertEqual(install_path('mod'), installpath_modules),  # via config file
         self.assertEqual(source_paths(), [testpath2])  # via command line
         self.assertEqual(build_path(), testpath1)  # via config file
-        self.assertEqual(get_repositorypath(), (os.path.join(topdir, 'ebfiles_repo'), 'somesubdir'))  # via config file
+        self.assertEqual(get_repositorypath(), [os.path.join(topdir, 'ebfiles_repo'), 'somesubdir'])  # via config file
 
         # hardcoded first entry
         self.assertEqual(options.robot_paths[0], '/tmp/foo')
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -6334,7 +6334,7 @@ def test_config_abs_path(self):
             r"^containerpath\s+\(F\) = /.*/test_topdir/test_middle_dir$",
             r"^installpath\s+\(E\) = /.*/test_topdir$",
             r"^prefix\s+\(C\) = /.*/test_topdir/test_middle_dir$",
-            r"^repositorypath\s+\(F\) = \('/apps/easyconfigs_archive', ' somesubdir'\)$",
+            r"^repositorypath\s+\(F\) = /apps/easyconfigs_archive,\s+somesubdir$",
             r"^sourcepath\s+\(C\) = /.*/test_topdir/test_middle_dir/test_subdir$",
             r"^robot-paths\s+\(E\) = /.*/test_topdir$",
         ]
@@ -6377,7 +6377,7 @@ def test_config_repositorypath(self):
         args[0] = '--repositorypath=%s,some/subdir' % repositorypath
         txt, _ = self._run_mock_eb(args, do_build=True, raise_error=True, testing=False, strip=True)
 
-        regex = re.compile(r"repositorypath\s+\(C\) = \('%s', 'some/subdir'\)" % repositorypath, re.M)
+        regex = re.compile(r"repositorypath\s+\(C\) = %s, some/subdir" % repositorypath, re.M)
         self.assertTrue(regex.search(txt), "Pattern '%s' should be found in: %s" % (regex.pattern, txt))
 
     # end-to-end testing of unknown filename