Repository: easybuilders/easybuild-framework

Issue:
nested temporary directories for easystack items
We use easystack files for our software stack, and I've run into weird issues several times where a build would fail due to what looked like an issue with the temporary directory. In all cases, the temporary directory looked like:
`/tmp/eb_something/eb_something/eb_something/....etc...`
and sometimes the error clearly stated that this path got too long.

It seems that each easystack item sets its own temporary directory as a subdirectory of the previous one, leading to this nested structure. While looking into it a bit more, I found that it's probably being caused by this call to `set_up_configuration`, which sets up the `eb_tmpdir`:
https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/main.py#L277
Adding a `print` there indeed shows the issue with a simple easystack:

```
$ cat test.yml 
easyconfigs:
  - GCCcore-11.2.0.eb
  - foss-2021b.eb
  - attr-2.5.1-GCCcore-11.2.0.eb
  - zlib-1.2.12.eb

$ $ eb -M --experimental --easystack ./test.yml 
== Temporary log file in case of crash /tmp/eb-5_bt16c0/easybuild-0qwq_ptk.log
eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

1 out of 61 required modules missing:

* pkgconf/1.8.0 (pkgconf-1.8.0.eb)

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow/eb-9m3qhlsm
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

eb_tmpdir = /tmp/eb-5_bt16c0/eb-rmkj0yno/eb-x1tle_ow/eb-9m3qhlsm/eb-u43uav4s
== found valid index for /home/bob/easybuild/cit-hpc-easybuild/easyconfigs, so using it...
== found valid index for /home/bob/easybuild/easybuild-easyconfigs/easybuild/easyconfigs, so using it...

No missing modules!

== Temporary log file(s) /tmp/eb-5_bt16c0/easybuild-0qwq_ptk.log* have been removed.
== Temporary directory /tmp/eb-5_bt16c0 has been removed.
```


Discussion:
Every item in the easystack file calls [set_up_configuration](https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/main.py#L277), which will set the tmpdir ([via the option parser?](https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/options.py#L1540)), and I guess this nesting then happens on lines 1878-1882 of https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/options.py#L1878. I'm not entirely sure yet why it happens, as the environment variables should be reset for every easystack item. However, inserting an `tempfile.tempdir = None` in the easystack loop [here](https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/main.py#L257) seems to solve the issue, so it definitely seems related to `tempfile` remembering the old `tmpdir`.
It only happens when `--tmpdir` is not set, by the way. But it does happen when `$TMPDIR` is defined.
The nested directory gets created here: https://github.com/easybuilders/easybuild-framework/blob/develop/easybuild/tools/options.py#L1860.
`os.environ['TMPDIR']` is not defined here, but `tempfile.tempdir` is set and points to the temp dir that was created before, leading to the nested structure.

I suspect it has to do with setting `tempfile.tempdir = None` and setting `TMPDIR` to the new temp dir, wich then leads to `tempfile.tempdir` being set to the new one later on as well (when e.g. `tempfile.mkdtemp()` is called). For any next easystack item, the os environment are restored/reset, but the `tempfile.tempdir` is not.
Reproducer based on what more or less happens in the actual code:

```
import os
import tempfile

def set_tmpdir():
    tmpdir = tempfile.mkdtemp(prefix="eb-")
    os.environ['TMPDIR'] = tmpdir
    tempfile.tempdir = None
    print("tempfile.gettempdir(): %s" % tempfile.gettempdir())
    return tmpdir

for i in range(3):
    if 'TMPDIR' in os.environ:
        del(os.environ['TMPDIR'])
    tempfile.tempdir = None
    tmpdir = set_tmpdir()
    print("new tmpdir: %s" % tmpdir)
    print("tempfile.tempdir: %s" % tempfile.tempdir)
```

This results in:
```
tempfile.gettempdir() = /tmp/eb-e5ihmqst
tmpdir = /tmp/eb-e5ihmqst
tempfile.tempdir = /tmp/eb-e5ihmqst
tempfile.gettempdir() = /tmp/eb-e5ihmqst/eb-p1snbzpj
tmpdir = /tmp/eb-e5ihmqst/eb-p1snbzpj
tempfile.tempdir = /tmp/eb-e5ihmqst/eb-p1snbzpj
tempfile.gettempdir() = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh
tmpdir = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh
tempfile.tempdir = /tmp/eb-e5ihmqst/eb-p1snbzpj/eb-otldk5xh
```

Uncommenting the `tempfile.tempdir = None` in the loop solves the issue:
```
tempfile.gettempdir(): /tmp/eb-carcmvfb
new tmpdir: /tmp/eb-carcmvfb
tempfile.tempdir: /tmp/eb-carcmvfb
tempfile.gettempdir(): /tmp/eb-7udoyzcp
new tmpdir: /tmp/eb-7udoyzcp
tempfile.tempdir: /tmp/eb-7udoyzcp
tempfile.gettempdir(): /tmp/eb-81phvqcx
new tmpdir: /tmp/eb-81phvqcx
tempfile.tempdir: /tmp/eb-81phvqcx
```

Alternatively, the `tempfile.tempdir = None` could also be added to the top of the `set_tmpdir` function.

Resolving patch:
diff --git a/easybuild/main.py b/easybuild/main.py
--- a/easybuild/main.py
+++ b/easybuild/main.py
@@ -42,6 +42,7 @@
 import os
 import stat
 import sys
+import tempfile
 import traceback
 
 # IMPORTANT this has to be the first easybuild import as it customises the logging
@@ -254,8 +255,9 @@ def process_easystack(easystack_path, args, logfile, testing, init_session_state
         easyconfig._easyconfigs_cache.clear()
         easyconfig._easyconfig_files_cache.clear()
 
-        # restore environment
+        # restore environment and reset tempdir (to avoid tmpdir path getting progressively longer)
         restore_env(init_env)
+        tempfile.tempdir = None
 
         # If EasyConfig specific arguments were supplied in EasyStack file
         # merge arguments with original command line args


Test patch:
diff --git a/test/framework/easystack.py b/test/framework/easystack.py
--- a/test/framework/easystack.py
+++ b/test/framework/easystack.py
@@ -31,6 +31,7 @@
 import os
 import re
 import sys
+import tempfile
 from unittest import TextTestRunner
 
 import easybuild.tools.build_log
@@ -129,11 +130,22 @@ def test_easystack_invalid_key2(self):
         self.assertErrorRegex(EasyBuildError, error_pattern, parse_easystack, test_easystack)
 
     def test_easystack_restore_env_after_each_build(self):
-        """Test that the build environment is reset for each easystack item"""
+        """Test that the build environment and tmpdir is reset for each easystack item"""
+
+        orig_tmpdir_tempfile = tempfile.gettempdir()
+        orig_tmpdir_env = os.getenv('TMPDIR')
+        orig_tmpdir_tempfile_len = len(orig_tmpdir_env.split(os.path.sep))
+        orig_tmpdir_env_len = len(orig_tmpdir_env.split(os.path.sep))
+
         test_es_txt = '\n'.join([
             "easyconfigs:",
             "  - toy-0.0-gompi-2018a.eb:",
             "  - libtoy-0.0.eb:",
+            # also include a couple of easyconfigs for which a module is already available in test environment,
+            # see test/framework/modules
+            "  - GCC-7.3.0-2.30",
+            "  - FFTW-3.3.7-gompi-2018a",
+            "  - foss-2018a",
         ])
         test_es_path = os.path.join(self.test_prefix, 'test.yml')
         write_file(test_es_path, test_es_txt)
@@ -143,10 +155,24 @@ def test_easystack_restore_env_after_each_build(self):
             '--easystack',
             test_es_path
         ]
-        stdout = self.eb_main(args, do_build=True, raise_error=True)
+        stdout = self.eb_main(args, do_build=True, raise_error=True, reset_env=False, redo_init_config=False)
         regex = re.compile(r"WARNING Loaded modules detected: \[.*gompi/2018.*\]\n")
         self.assertFalse(regex.search(stdout), "Pattern '%s' should not be found in: %s" % (regex.pattern, stdout))
 
+        # temporary directory after run should be exactly 2 levels deeper than original one:
+        # - 1 level added by setting up configuration in EasyBuild main function
+        # - 1 extra level added by first re-configuration for easystack item
+        #   (because $TMPDIR set by configuration done in main function is retained)
+        tmpdir_tempfile = tempfile.gettempdir()
+        tmpdir_env = os.getenv('TMPDIR')
+        tmpdir_tempfile_len = len(tmpdir_tempfile.split(os.path.sep))
+        tmpdir_env_len = len(tmpdir_env.split(os.path.sep))
+
+        self.assertEqual(tmpdir_tempfile_len, orig_tmpdir_tempfile_len + 2)
+        self.assertEqual(tmpdir_env_len, orig_tmpdir_env_len + 2)
+        self.assertTrue(tmpdir_tempfile.startswith(orig_tmpdir_tempfile))
+        self.assertTrue(tmpdir_env.startswith(orig_tmpdir_env))
+
     def test_missing_easyconfigs_key(self):
         """Test that EasyStack file that doesn't contain an EasyConfigs key will fail with sane error message"""
         topdir = os.path.dirname(os.path.abspath(__file__))