Repository: spack/spack

Issue:
Installation issue: openjdk
### Steps to reproduce the issue

```
$ spack install openjdk@11.0.8_10
==> Warning: gcc@8.4.1 cannot build optimized binaries for "cascadelake". Using best target possible: "skylake_avx512"
==> Installing openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd
==> No binary for openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd found: installing from source
==> Using cached archive: /g/data/jh2/bw0729/spack/var/spack/cache/_source-cache/archive/6e/6e4cead158037cb7747ca47416474d4f408c9126be5b96f9befd532e0a762b47.tar.gz
==> No patches needed for openjdk
==> openjdk: Executing phase: 'install'
==> Error: PermissionError: [Errno 13] Permission denied: '/g/data/jh2/bw0729/spack/opt/spack/linux-centos8-skylake_avx512/gcc-8.4.1/openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd/legal/java.smartcardio/pcsclite.md'

/g/data/jh2/bw0729/spack/var/spack/repos/builtin/packages/openjdk/package.py:123, in install:
        121    def install(self, spec, prefix):
        122        top_dir = 'Contents/Home/' if platform.system() == "Darwin" else '.'
  >>    123        install_tree(top_dir, prefix)

See build log for details:
  /scratch/jh2/bw0729/tmp/spack-stage/spack-stage-openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd/spack-build-out.txt
```

### Information on your system

* **Spack:** 0.16.2-3870-5916afec84
* **Python:** 3.6.8
* **Platform:** linux-centos8-cascadelake
* **Concretizer:** original

### Additional information

[spack-build-out.txt](https://github.com/spack/spack/files/6965940/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/6965941/spack-build-env.txt)


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate
Installation issue: openjdk
### Steps to reproduce the issue

```
$ spack install openjdk@11.0.8_10
==> Warning: gcc@8.4.1 cannot build optimized binaries for "cascadelake". Using best target possible: "skylake_avx512"
==> Installing openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd
==> No binary for openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd found: installing from source
==> Using cached archive: /g/data/jh2/bw0729/spack/var/spack/cache/_source-cache/archive/6e/6e4cead158037cb7747ca47416474d4f408c9126be5b96f9befd532e0a762b47.tar.gz
==> No patches needed for openjdk
==> openjdk: Executing phase: 'install'
==> Error: PermissionError: [Errno 13] Permission denied: '/g/data/jh2/bw0729/spack/opt/spack/linux-centos8-skylake_avx512/gcc-8.4.1/openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd/legal/java.smartcardio/pcsclite.md'

/g/data/jh2/bw0729/spack/var/spack/repos/builtin/packages/openjdk/package.py:123, in install:
        121    def install(self, spec, prefix):
        122        top_dir = 'Contents/Home/' if platform.system() == "Darwin" else '.'
  >>    123        install_tree(top_dir, prefix)

See build log for details:
  /scratch/jh2/bw0729/tmp/spack-stage/spack-stage-openjdk-11.0.8_10-ezp6p5txswxryvypbhmw7zsqm45yn2kd/spack-build-out.txt
```

### Information on your system

* **Spack:** 0.16.2-3870-5916afec84
* **Python:** 3.6.8
* **Platform:** linux-centos8-cascadelake
* **Concretizer:** original

### Additional information

[spack-build-out.txt](https://github.com/spack/spack/files/6965940/spack-build-out.txt)
[spack-build-env.txt](https://github.com/spack/spack/files/6965941/spack-build-env.txt)


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have run `spack maintainers <name-of-the-package>` and **@mentioned** any maintainers
- [X] I have uploaded the build log and environment files
- [X] I have searched the issues of this repo and believe this is not a duplicate


Discussion:
@BenWibking You have the strangest bug reports. :-)

I don't know if this is something that anyone can really help you
with.  The openjdk package is very simple: dowload tar file, unpack
and copy into place (the python version of tar-copy).

The error message is permission denied.  So, walk the path for the
install directory and check each step.  Check if you can manually cd
to the install dir and touch a file.

I'm guessing /g/data/jh2/bw0729 is a scratch directory?
Check that the mount point is read-write.
Check that the directories are either owned by you or group writable.
Maybe try something in your home directory.
Check that the file system isn't full.

If nothing obvious works, you'll probably need to check with your
local sysadmin.
@BenWibking Did you ever get this to work?
Not with Spack, no. The sysadmins managed to install openjdk+hpcviewer+hpctoolkit manually, though.
Ok, I figured out what the issue was. It turns out there is a bug in python versions < 3.7.4 that causes extended attributes used by Lustre to cause PermissionError when using shutil.copy2 (which is used by https://github.com/spack/spack/blob/develop/lib/spack/llnl/util/filesystem.py#L504 to copy directory trees).

Setting SPACK_PYTHON to point to a python executable >= version 3.7.4 fixes the problem. However, Spack should be fixed for python versions below this, or at least issue a warning that this should be expected to fail on Lustre!
The upstream Python bug is: https://bugs.python.org/issue24538 and the confirmation that it affects Lustre is here: https://bugs.python.org/issue24538#msg301493
@BenWibking You have the strangest bug reports. :-)

I don't know if this is something that anyone can really help you
with.  The openjdk package is very simple: dowload tar file, unpack
and copy into place (the python version of tar-copy).

The error message is permission denied.  So, walk the path for the
install directory and check each step.  Check if you can manually cd
to the install dir and touch a file.

I'm guessing /g/data/jh2/bw0729 is a scratch directory?
Check that the mount point is read-write.
Check that the directories are either owned by you or group writable.
Maybe try something in your home directory.
Check that the file system isn't full.

If nothing obvious works, you'll probably need to check with your
local sysadmin.
@BenWibking Did you ever get this to work?
Not with Spack, no. The sysadmins managed to install openjdk+hpcviewer+hpctoolkit manually, though.
Ok, I figured out what the issue was. It turns out there is a bug in python versions < 3.7.4 that causes extended attributes used by Lustre to cause PermissionError when using shutil.copy2 (which is used by https://github.com/spack/spack/blob/develop/lib/spack/llnl/util/filesystem.py#L504 to copy directory trees).

Setting SPACK_PYTHON to point to a python executable >= version 3.7.4 fixes the problem. However, Spack should be fixed for python versions below this, or at least issue a warning that this should be expected to fail on Lustre!
The upstream Python bug is: https://bugs.python.org/issue24538 and the confirmation that it affects Lustre is here: https://bugs.python.org/issue24538#msg301493

Resolving patch:
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -84,6 +84,74 @@
     "visit_directory_tree",
 ]
 
+if sys.version_info < (3, 7, 4):
+    # monkeypatch shutil.copystat to fix PermissionError when copying read-only
+    # files on Lustre when using Python < 3.7.4
+
+    def copystat(src, dst, follow_symlinks=True):
+        """Copy file metadata
+        Copy the permission bits, last access time, last modification time, and
+        flags from `src` to `dst`. On Linux, copystat() also copies the "extended
+        attributes" where possible. The file contents, owner, and group are
+        unaffected. `src` and `dst` are path names given as strings.
+        If the optional flag `follow_symlinks` is not set, symlinks aren't
+        followed if and only if both `src` and `dst` are symlinks.
+        """
+
+        def _nop(args, ns=None, follow_symlinks=None):
+            pass
+
+        # follow symlinks (aka don't not follow symlinks)
+        follow = follow_symlinks or not (os.path.islink(src) and os.path.islink(dst))
+        if follow:
+            # use the real function if it exists
+            def lookup(name):
+                return getattr(os, name, _nop)
+
+        else:
+            # use the real function only if it exists
+            # *and* it supports follow_symlinks
+            def lookup(name):
+                fn = getattr(os, name, _nop)
+                if sys.version_info >= (3, 3):
+                    if fn in os.supports_follow_symlinks:  # novermin
+                        return fn
+                return _nop
+
+        st = lookup("stat")(src, follow_symlinks=follow)
+        mode = stat.S_IMODE(st.st_mode)
+        lookup("utime")(dst, ns=(st.st_atime_ns, st.st_mtime_ns), follow_symlinks=follow)
+
+        # We must copy extended attributes before the file is (potentially)
+        # chmod()'ed read-only, otherwise setxattr() will error with -EACCES.
+        shutil._copyxattr(src, dst, follow_symlinks=follow)
+
+        try:
+            lookup("chmod")(dst, mode, follow_symlinks=follow)
+        except NotImplementedError:
+            # if we got a NotImplementedError, it's because
+            #   * follow_symlinks=False,
+            #   * lchown() is unavailable, and
+            #   * either
+            #       * fchownat() is unavailable or
+            #       * fchownat() doesn't implement AT_SYMLINK_NOFOLLOW.
+            #         (it returned ENOSUP.)
+            # therefore we're out of options--we simply cannot chown the
+            # symlink.  give up, suppress the error.
+            # (which is what shutil always did in this circumstance.)
+            pass
+        if hasattr(st, "st_flags"):
+            try:
+                lookup("chflags")(dst, st.st_flags, follow_symlinks=follow)
+            except OSError as why:
+                for err in "EOPNOTSUPP", "ENOTSUP":
+                    if hasattr(errno, err) and why.errno == getattr(errno, err):
+                        break
+                else:
+                    raise
+
+    shutil.copystat = copystat
+
 
 def getuid():
     if is_windows:


Test patch: