internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: easybuilders/easybuild-framework

Issue:
dry run fails on re.compile when software name contains metacharacters like '++'
e.g. 

```
# eb Blitz++-0.10-foss-2016a.eb -x
(...)
ERROR: Traceback (most recent call last):
  File "easybuild-framework/easybuild/main.py", line 116, in build_and_install_software
    (ec_res['success'], app_log, err) = build_and_install_one(ec, init_env)
  File "easybuild-framework/easybuild/framework/easyblock.py", line 2487, in build_and_install_one
    app = app_class(ecdict['ec'])
  File "easybuild-framework/easybuild/framework/easyblock.py", line 244, in __init__
    self.init_dry_run()
  File "easybuild-framework/easybuild/framework/easyblock.py", line 291, in init_dry_run
    dry_run_set_dirs(tmp_root_dir, self.builddir, self.installdir, self.installdir_mod)
  File "easybuild-framework/easybuild/tools/build_log.py", line 245, in dry_run_set_dirs
    DRY_RUN_SOFTWARE_INSTALL_DIR = (re.compile(software_installdir), software_installdir[len(prefix):])
  File "/usr/lib64/python2.6/re.py", line 190, in compile
    return _compile(pattern, flags)
  File "/usr/lib64/python2.6/re.py", line 245, in _compile
    raise error, v # invalid expression
error: multiple repeat
```

needs `re.escape`?


Discussion:
@migueldiascosta Nice catch...

`re.escape` seems like the easiest/best way to fix this problem, yes

Are you up for issuing a PR for that?
@boegel ok, I'll replace `re.compile(software_installdir)` with `re.compile(re.escape(software_installdir))` 

`re.compile(builddir)` seems to be safe by construction (`remove_unwanted_chars` in `gen_builddir`)

regarding `re.compile(module_installdir)`, I suppose one wouldn't expect those characters in `module_installdir`, but shall I escape it anyway to be safe?
@migueldiascosta I'd keep things consistent and use `re.compile(re.escape(...))` in all three cases, it doesn't hurt (just keep in mind to stick to lines of <= 120 chars)

Resolving patch:
diff --git a/easybuild/tools/build_log.py b/easybuild/tools/build_log.py
--- a/easybuild/tools/build_log.py
+++ b/easybuild/tools/build_log.py
@@ -236,13 +236,13 @@ def dry_run_set_dirs(prefix, builddir, software_installdir, module_installdir):
     :param module_installdir: fake module install directory
     """
     global DRY_RUN_BUILD_DIR
-    DRY_RUN_BUILD_DIR = (re.compile(builddir), builddir[len(prefix):])
+    DRY_RUN_BUILD_DIR = (re.compile(re.escape(builddir)), builddir[len(prefix):])
 
     global DRY_RUN_MODULES_INSTALL_DIR
-    DRY_RUN_MODULES_INSTALL_DIR = (re.compile(module_installdir), module_installdir[len(prefix):])
+    DRY_RUN_MODULES_INSTALL_DIR = (re.compile(re.escape(module_installdir)), module_installdir[len(prefix):])
 
     global DRY_RUN_SOFTWARE_INSTALL_DIR
-    DRY_RUN_SOFTWARE_INSTALL_DIR = (re.compile(software_installdir), software_installdir[len(prefix):])
+    DRY_RUN_SOFTWARE_INSTALL_DIR = (re.compile(re.escape(software_installdir)), software_installdir[len(prefix):])
 
 
 def dry_run_msg(msg, silent=False):


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
--new/update-pr fails in extended dry run mode when adding a patch file
When running `--update-pr -x` and adding a new patch file, it fails saying it cannot find the patch file, which should have been copied to the temp dir. But the actual `--update-pr` works, so `-x` must be disabling the copy?

Additionally, doing the same to submit changes to a patch, dry run mode doesn't show the changes to the patch file, although the actual `--update-pr` does submit them.


Discussion:
just ran into this issue again, this time with `--new-pr`
I can submit a PR for this, but I see two alternatives:

1) skip staging the patch in `_easyconfigs_pr_common` when in extended dry run mode (fixes the crash but will still not show the patch)

2) add `force_in_dry_run=True` to `copy_file` in `copy_patch_files`, as in `copy_easyconfigs` (not sure if it will affect anything else)

Resolving patch:
diff --git a/easybuild/framework/easyconfig/easyconfig.py b/easybuild/framework/easyconfig/easyconfig.py
--- a/easybuild/framework/easyconfig/easyconfig.py
+++ b/easybuild/framework/easyconfig/easyconfig.py
@@ -1579,7 +1579,7 @@ def copy_patch_files(patch_specs, target_dir):
     }
     for patch_path, soft_name in patch_specs:
         target_path = det_location_for(patch_path, target_dir, soft_name, os.path.basename(patch_path))
-        copy_file(patch_path, target_path)
+        copy_file(patch_path, target_path, force_in_dry_run=True)
         patched_files['paths_in_repo'].append(target_path)
 
     return patched_files


Test patch:
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -2558,6 +2558,19 @@ def test_new_update_pr(self):
             regex = re.compile(regex, re.M)
             self.assertTrue(regex.search(txt), "Pattern '%s' found in: %s" % (regex.pattern, txt))
 
+        # should also work with a patch
+        args.append(toy_patch)
+        self.mock_stdout(True)
+        self.eb_main(args, do_build=True, raise_error=True, testing=False)
+        txt = self.get_stdout()
+        self.mock_stdout(False)
+
+        regexs[-2] = r"^\s*3 files changed"
+        regexs.append(r".*_fix-silly-typo-in-printf-statement.patch\s*\|")
+        for regex in regexs:
+            regex = re.compile(regex, re.M)
+            self.assertTrue(regex.search(txt), "Pattern '%s' found in: %s" % (regex.pattern, txt))
+
         # modifying an existing easyconfig requires a custom PR title
         gcc_ec = os.path.join(test_ecs, 'g', 'GCC', 'GCC-4.9.2.eb')
         self.assertTrue(os.path.exists(gcc_ec))
Repository: easybuilders/easybuild-framework

Issue:
Dumping parsed easyconfig files
With #2562 we introduce the capability of making modifications to the easyconfig file that is read from disk in a (potentially) non-reproducible way.

At https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/framework/easyblock.py#L2869 there is already a feature that dumps the final parsed easyconfig to the installation directory (in a `reprod` subdir).  This was originally introduced for minimal toolchains but since (some level of) subtoolchain resolution is now the default, and because of the new hook, this should be done more widely.


Discussion:
FYI, the file in the reprod contains an easyconfig with the changes in place. I just checked.
I.e., changes done by the hooks.
I expected so, the issue is just then in what cases do we trigger that dump, currently it's only done if you use minimal toolchains but given that subtoolchain resolution is now the default, I don't really see a good reason not to do it in all cases...especially since it does zero harm and saves us from having to create complicated logic.

Resolving patch:
diff --git a/easybuild/framework/easyblock.py b/easybuild/framework/easyblock.py
--- a/easybuild/framework/easyblock.py
+++ b/easybuild/framework/easyblock.py
@@ -2866,13 +2866,14 @@ def build_and_install_one(ecdict, init_env):
             buildstats = get_build_stats(app, start_time, build_option('command_line'))
             _log.info("Build stats: %s" % buildstats)
 
-            if build_option("minimal_toolchains"):
-                # for reproducability we dump out the parsed easyconfig since the contents are affected when
-                # --minimal-toolchains (and --use-existing-modules) is used
-                # TODO --try-toolchain needs to be fixed so this doesn't play havoc with it's usability
-                reprod_spec = os.path.join(new_log_dir, 'reprod', ec_filename)
+            # for reproducability we dump out the fully processed easyconfig since the contents can be affected
+            # by subtoolchain resolution (and related options) and/or hooks
+            reprod_spec = os.path.join(new_log_dir, 'reprod', ec_filename)
+            try:
                 app.cfg.dump(reprod_spec)
-                _log.debug("Dumped easyconfig tweaked via --minimal-toolchains to %s", reprod_spec)
+                _log.info("Dumped fully processed easyconfig to %s", reprod_spec)
+            except NotImplementedError as err:
+                _log.warn("Unable to dumped fully processed easyconfig to %s: %s", reprod_spec, err)
 
             try:
                 # upload easyconfig (and patch files) to central repository


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
setting CDPATH breaks libxsmm build
If you have `CDPATH` environmental variable, it breaks the build of `libxsmm`. (version 1.8.3 confirmed, second run, when shared objects created from static libraries). It is probably caused by `patsubst` make function or automake or something else, as issuing `cd` command might generate an output if `CDPATH` is set. It makes sense to unset `CDPATH` as it might cause problems in other places as well.


Discussion:
Confirmed here, simply setting `$CDPATH` to something trivial like `.:/tmp` is enough to cause trouble (e.g. break the installation of `libxsmm-1.8.3-foss-2018a.eb`).

We should simply always unset `$CDPATH` if it is defined, there's no benefit that `eb` can have from having it set...

Resolving patch:
diff --git a/easybuild/main.py b/easybuild/main.py
--- a/easybuild/main.py
+++ b/easybuild/main.py
@@ -187,6 +187,11 @@ def main(args=None, logfile=None, do_build=None, testing=False, modtool=None):
     :param do_build: whether or not to actually perform the build
     :param testing: enable testing mode
     """
+    # if $CDPATH is set, unset it, it'll only cause trouble...
+    # see https://github.com/easybuilders/easybuild-framework/issues/2944
+    if 'CDPATH' in os.environ:
+        del os.environ['CDPATH']
+
     # purposely session state very early, to avoid modules loaded by EasyBuild meddling in
     init_session_state = session_state()
 


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
try to avoid requiring setuptools to install EasyBuild
cfr. #2428


Discussion:

Resolving patch:
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -72,7 +72,7 @@
 from easybuild.tools.docs import avail_toolchain_opts, avail_easyconfig_params, avail_easyconfig_templates
 from easybuild.tools.docs import list_easyblocks, list_toolchains
 from easybuild.tools.environment import restore_env, unset_env_vars
-from easybuild.tools.filetools import CHECKSUM_TYPE_SHA256, CHECKSUM_TYPES, install_fake_vsc
+from easybuild.tools.filetools import CHECKSUM_TYPE_SHA256, CHECKSUM_TYPES, install_fake_vsc, which
 from easybuild.tools.github import GITHUB_EB_MAIN, GITHUB_EASYCONFIGS_REPO
 from easybuild.tools.github import GITHUB_PR_DIRECTION_DESC, GITHUB_PR_ORDER_CREATED, GITHUB_PR_STATE_OPEN
 from easybuild.tools.github import GITHUB_PR_STATES, GITHUB_PR_ORDERS, GITHUB_PR_DIRECTIONS
@@ -1422,9 +1422,18 @@ def parse_external_modules_metadata(cfgs):
 
     # use external modules metadata configuration files that are available by default, unless others are specified
     if not cfgs:
-        # we expect to find *external_modules_metadata.cfg files in etc/
-        topdir = os.path.dirname(os.path.dirname(os.path.dirname(__file__)))
-        cfgs = glob.glob(os.path.join(topdir, 'etc', '*external_modules_metadata.cfg'))
+        cfgs = []
+
+        # we expect to find *external_modules_metadata.cfg files in etc/ on same level as easybuild/framework
+        topdirs = [os.path.dirname(os.path.dirname(os.path.dirname(__file__)))]
+
+        # etc/ could also be located next to bin/
+        eb_cmd = which('eb')
+        if eb_cmd:
+            topdirs.append(os.path.dirname(os.path.dirname(eb_cmd)))
+
+        for topdir in topdirs:
+            cfgs.extend(glob.glob(os.path.join(topdir, 'etc', '*external_modules_metadata.cfg')))
 
         if cfgs:
             _log.info("Using default external modules metadata cfg files: %s", cfgs)
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -33,6 +33,7 @@
 import glob
 import os
 from distutils import log
+from distutils.core import setup
 
 from easybuild.tools.version import VERSION
 
@@ -47,13 +48,6 @@ def read(fname):
 # log levels: 0 = WARN (default), 1 = INFO, 2 = DEBUG
 log.set_verbosity(1)
 
-try:
-    from setuptools import setup
-    log.info("Installing with setuptools.setup...")
-except ImportError as err:
-    log.info("Failed to import setuptools.setup, so falling back to distutils.setup")
-    from distutils.core import setup
-
 log.info("Installing version %s (API version %s)" % (VERSION, API_VERSION))
 
 
@@ -123,13 +117,4 @@ def find_rel_test():
         "Topic :: Software Development :: Build Tools",
     ],
     platforms="Linux",
-    test_suite="test.framework.suite",
-    zip_safe=False,
-    extras_require={
-        'yeb': ["PyYAML >= 3.11"],
-        'coloredlogs': [
-            'coloredlogs',
-            'humanfriendly',  # determine whether terminal supports ANSI color
-        ],
-    },
 )


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
inject-checksum tries mkdir on the installdir
This shows the problem:
```
b-an03 [EMAN2]$ pwd
/home/a/ake/Projects/git-repos/easybuild/easybuild_common/custom/easyconfigs/e/EMAN2
b-an03 [EMAN2]$ eb --version
This is EasyBuild 4.0.1 (framework: 4.0.1, easyblocks: 4.0.1) on host b-an03.hpc2n.umu.se.
b-an03 [EMAN2]$ eb EMAN2-2.3-fosscuda-2019a-Python-2.7.15.eb --inject-checksums --hooks="" --robot-paths=/home/a/ake/Projects/git-repos/easybuild/easybuild_common/custom/easyconfigs:/hpc2n/eb/custom/easyconfigs:/hpc2n/eb/software/Core/EasyBuild/4.0.1/easybuild/easyconfigs
== temporary log file in case of crash /scratch/eb-ake-tmp/eb-ncNnn3/easybuild-S9I5Wr.log
== injecting sha256 checksums in /afs/hpc2n.umu.se/home/a/ake/Projects/git-repos/easybuild/easybuild_common/custom/easyconfigs/e/EMAN2/EMAN2-2.3-fosscuda-2019a-Python-2.7.15.eb
== fetching sources & patches for EMAN2-2.3-fosscuda-2019a-Python-2.7.15.eb...
ERROR: Failed to create directory /hpc2n/eb/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/EMAN2/2.3-Python-2.7.15: [Errno 30] Read-only file system: '/hpc2n/eb/software/MPI/GCC-CUDA/8.2.0-2.31.1-10.1.105/OpenMPI/3.1.3/EMAN2'
```


Discussion:
I suppose `--inject-checksums` (and anything else running `fetch_step` without intending to install) should set `pre_create_installdir` to False? 

https://github.com/easybuilders/easybuild-framework/blob/b06b84e58368db2baabda4c37ed2806e4a778778/easybuild/framework/easyblock.py#L1783-L1789
Yeah, that is likely a good idea.

Resolving patch:
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -989,6 +989,10 @@ def _postprocess_config(self):
             self.options.ignore_osdeps = True
             self.options.modules_tool = None
 
+        # imply --disable-pre-create-installdir with --inject-checksums
+        if self.options.inject_checksums:
+            self.options.pre_create_installdir = False
+
     def _postprocess_list_avail(self):
         """Create all the additional info that can be requested (exit at the end)"""
         msg = ''


Test patch:
diff --git a/test/framework/options.py b/test/framework/options.py
--- a/test/framework/options.py
+++ b/test/framework/options.py
@@ -3900,6 +3900,9 @@ def test_inject_checksums(self):
         self.mock_stdout(False)
         self.mock_stderr(False)
 
+        # make sure software install directory is *not* created (see bug issue #3064)
+        self.assertFalse(os.path.exists(os.path.join(self.test_installpath, 'software', 'toy')))
+
         # SHA256 is default type of checksums used
         self.assertTrue("injecting sha256 checksums in" in stdout)
         self.assertEqual(stderr, '')
Repository: easybuilders/easybuild-framework

Issue:
"Failed to delete test branch from GitHub" when using --check-github
When setting EASYBUILD_GIT_WORKING_DIRS_PATH to my local repo I get the following error:

```
* push access to Flamefire/easybuild-easyconfigs repo @ GitHub...OK
WARNNIG: failed to delete test branch from GitHub: Cmd('git') failed due to: exit code(1)
  cmdline: git push --delete --porcelain origin test_branch_ZOSYM
  stderr: 'error: unable to delete 'test_branch_ZOSYM': remote ref does not exist
error: failed to push some refs to '/home/s3248973/.local/EasyBuildDev/easybuild-easyconfigs/.git''
```

When I go to the mentioned folder (without the .git) and execute the command it does work


Discussion:

Resolving patch:
diff --git a/easybuild/tools/github.py b/easybuild/tools/github.py
--- a/easybuild/tools/github.py
+++ b/easybuild/tools/github.py
@@ -1624,6 +1624,8 @@ def check_github():
     * check whether creating gists works
     * check whether location to local working directories for Git repositories is available (not strictly needed)
     """
+    debug = build_option('debug')
+
     # start by assuming that everything works, individual checks will disable action that won't work
     status = {}
     for action in ['--from-pr', '--new-pr', '--review-pr', '--upload-test-report', '--update-pr']:
@@ -1716,9 +1718,9 @@ def check_github():
     git_repo, res, push_err = None, None, None
     branch_name = 'test_branch_%s' % ''.join(random.choice(ascii_letters) for _ in range(5))
     try:
-        git_repo = init_repo(git_working_dir, GITHUB_EASYCONFIGS_REPO, silent=True)
+        git_repo = init_repo(git_working_dir, GITHUB_EASYCONFIGS_REPO, silent=not debug)
         remote_name = setup_repo(git_repo, github_account, GITHUB_EASYCONFIGS_REPO, 'master',
-                                 silent=True, git_only=True)
+                                 silent=not debug, git_only=True)
         git_repo.create_head(branch_name)
         res = getattr(git_repo.remotes, remote_name).push(branch_name)
     except Exception as err:
@@ -1749,12 +1751,11 @@ def check_github():
     print_msg(check_res, log=_log, prefix=False)
 
     # cleanup: delete test branch that was pushed to GitHub
-    if git_repo:
+    if git_repo and push_err is None:
         try:
-            if git_repo and hasattr(git_repo, 'remotes') and hasattr(git_repo.remotes, 'origin'):
-                git_repo.remotes.origin.push(branch_name, delete=True)
+            getattr(git_repo.remotes, remote_name).push(branch_name, delete=True)
         except GitCommandError as err:
-            sys.stderr.write("WARNNIG: failed to delete test branch from GitHub: %s\n" % err)
+            sys.stderr.write("WARNING: failed to delete test branch from GitHub: %s\n" % err)
 
     # test creating a gist
     print_msg("* creating gists...", log=_log, prefix=False, newline=False)


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
--test-update-deps + --try-toolchain fails with Perl versionsuffix
`eb --try-toolchain=intel,2019b --robot --experimental --try-update-deps BioPerl-1.7.2-intel-2018b-Perl-5.28.0.eb`
Gives this with minimal-toolchains=False: 
`ERROR: Missing dependencies: XML-LibXML/2.0132-intel-2019b-Perl-5.30.0`
And this with minimal-toolchains=True:
`Failed to determine minimal toolchain for dep XML-LibXML 2.0132-Perl-5.30.0`

I think the expected behavior should be to find XML-LibXML-2.0201-GCCcore-8.3.0.eb, as this is compatible with intel/2019b.


Discussion:

Resolving patch:
diff --git a/easybuild/framework/easyconfig/tweak.py b/easybuild/framework/easyconfig/tweak.py
--- a/easybuild/framework/easyconfig/tweak.py
+++ b/easybuild/framework/easyconfig/tweak.py
@@ -961,6 +961,7 @@ def map_easyconfig_to_target_tc_hierarchy(ec_spec, toolchain_mapping, targetdir=
     if update_dep_versions and (list_deps_versionsuffixes(ec_spec) or parsed_ec['versionsuffix']):
         # We may need to update the versionsuffix if it is like, for example, `-Python-2.7.8`
         versonsuffix_mapping = map_common_versionsuffixes('Python', parsed_ec['toolchain'], toolchain_mapping)
+        versonsuffix_mapping.update(map_common_versionsuffixes('Perl', parsed_ec['toolchain'], toolchain_mapping))
 
     if update_build_specs is not None:
         if 'version' in update_build_specs:


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
--cuda-compute-capabilites help should show desired format
Since there are quite a number of ways that CUDA capabilities are defined (e.g. `sm_75`, `75` and `7.5` define the same level), it would be nice if the help message for `--cuda-compute-capabilites` mentioned the desired format.


Discussion:

Resolving patch:
diff --git a/easybuild/tools/options.py b/easybuild/tools/options.py
--- a/easybuild/tools/options.py
+++ b/easybuild/tools/options.py
@@ -356,8 +356,9 @@ def override_options(self):
             'consider-archived-easyconfigs': ("Also consider archived easyconfigs", None, 'store_true', False),
             'containerize': ("Generate container recipe/image", None, 'store_true', False, 'C'),
             'copy-ec': ("Copy specified easyconfig(s) to specified location", None, 'store_true', False),
-            'cuda-compute-capabilities': ("List of CUDA compute capabilities to use when building GPU software",
-                                          'strlist', 'extend', None),
+            'cuda-compute-capabilities': ("List of CUDA compute capabilities to use when building GPU software; "
+                                          "values should be specified as digits separated by a dot, "
+                                          "for example: 3.5,5.0,7.2", 'strlist', 'extend', None),
             'debug-lmod': ("Run Lmod modules tool commands in debug module", None, 'store_true', False),
             'default-opt-level': ("Specify default optimisation level", 'choice', 'store', DEFAULT_OPT_LEVEL,
                                   Compiler.COMPILER_OPT_FLAGS),


Test patch:
Repository: easybuilders/easybuild-framework

Issue:
Containers still being configured for EB 3.x
Containers from EB 4.x are still being configured for 3.9:
```
# install EasyBuild using pip
pip install -U setuptools
pip install 'vsc-install<0.11.4' 'vsc-base<2.9.0'
pip install easybuild
```


Discussion:

Resolving patch:
diff --git a/easybuild/tools/containers/singularity.py b/easybuild/tools/containers/singularity.py
--- a/easybuild/tools/containers/singularity.py
+++ b/easybuild/tools/containers/singularity.py
@@ -314,13 +314,6 @@ def resolve_template_data(self):
                 "# install EasyBuild using pip",
                 # upgrade pip
                 "pip install -U pip",
-                "pip install wheel",
-                # EasyBuild 3.x requires setuptools as runtime dependency
-                "pip install -U setuptools",
-                # stick to previous version of vsc-install to avoid requiring mock (which causes installation problems)
-                # stick to previous version of vsc-base to avoid requiring 'future' (irrelevant for EasyBuild)
-                # this is just a temporary measure, since vsc-install & vsc-base have been ingested for EasyBuild 4.x
-                "pip install 'vsc-install<0.11.4' 'vsc-base<2.9.0'",
                 "pip install easybuild",
             ])
 


Test patch:
diff --git a/test/framework/containers.py b/test/framework/containers.py
--- a/test/framework/containers.py
+++ b/test/framework/containers.py
@@ -141,9 +141,7 @@ def test_end2end_singularity_recipe_config(self):
             self.assertTrue(regex.search(txt), "Pattern '%s' found in: %s" % (regex.pattern, txt))
 
         pip_patterns = [
-            # EasyBuild and dependencies are installed with pip by default
-            "pip install -U setuptools",
-            "pip install.*vsc-base",
+            # EasyBuild is installed with pip by default
             "pip install easybuild",
         ]
         post_commands_patterns = [