Repository: spack/spack

Issue:
Cray shasta systems: Spack cannot detect architecture
My team is enthusiastic to give Spack a chance to build packages for a pair of large Cray Shasta systems coming later this year.  On our test system, I am experiencing issues with what I think are the Spack initialization and craype recognition routines (possibly the cle-release file .... I was looking at issue [12178](https://github.com/spack/spack/issues/12187)) to come to this conclusion.

Spack fails to start normally if the module for craype is loaded.  The cle-release file on Shasta in UAN (front-end node), UAI (login container, and CLE (Compute node) are all identical:
```
cat /etc/opt/cray/release/cle-release
PRODUCT="Cray's Linux Environment"
OS=SLES15SP1
ARCH=x86_64
VERSION=1.2.0
DATE=20200421195827
```
spack debug fails or I'd provide it.

### Error Message

```console
# . $SPACK_ROOT/share/spack/setup-env.sh
Traceback (most recent call last):
  File "/apps/dev/spack/current/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/main.py", line 711, in main
    print_setup_info(*args.print_shell_vars.split(','))
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/main.py", line 628, in print_setup_info
    shell_set('_sp_sys_type', spack.architecture.sys_type())
  File "/apps/dev/spack/spack-0.14.2/lib/spack/llnl/util/lang.py", line 178, in _memoized_function
    func.cache[args] = func(*args)
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/architecture.py", line 532, in sys_type
    arch = Arch(platform(), 'default_os', 'default_target')
  File "/apps/dev/spack/spack-0.14.2/lib/spack/llnl/util/lang.py", line 178, in _memoized_function
    func.cache[args] = func(*args)
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/architecture.py", line 516, in platform
    return platform_cls()
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/platforms/cray.py", line 76, in __init__
    back_distro = Cnl()
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/operating_systems/cnl.py", line 81, in __init__
    version = self._detect_crayos_version()
  File "/apps/dev/spack/spack-0.14.2/lib/spack/spack/operating_systems/cnl.py", line 92, in _detect_crayos_version
    v = spack.version.Version(release_attrs['RELEASE'])
```

### Information on your system



<!-- Please include the output of `spack debug report` -->

Spack

<!-- If you have any relevant configuration detail (custom `packages.yaml` or `modules.yaml`, etc.) you can add that here as well. -->

### Additional information

<!-- These boxes can be checked by replacing [ ] with [x] or by clicking them after submitting the issue. -->
- [ ] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [x] I have searched the issues of this repo and believe this is not a duplicate
- [ ] I have run the failing commands in debug mode and reported the output

<!-- We encourage you to try, as much as possible, to reduce your problem to the minimal example that still reproduces the issue. That would help us a lot in fixing it quickly and effectively!

If you want to ask a question about the tool (how to use it, what it can currently do, etc.), try the `#general` channel on our Slack first. We have a welcoming community and chances are you'll get your reply faster and without opening an issue.

Other than that, thanks for taking the time to contribute to Spack! -->



Discussion:
Can you try running from the develop branch? [Several recent PRs](https://github.com/spack/spack/pulls?q=is%3Apr+is%3Aclosed+label%3Acray) have been merged to fix Cray support. These will be included in the next release (0.15) which is expected to come out any day now. @becker33 
I'm working on the release right now, ETA tonight. This issue sounds like it should be resolved by #12989, which is in the 0.15.0 release.
I installed v0.15 today.  I'm still having problems. Running the initialization, it looks like something is still broken and it doesn't recognize my platform.  Oddly, at v0.14, I could at least unload craype and not get this error (see below this error.)

ln01-nmn:/apps/spack/spack-0.15.0/share/spack # . ./setup-env.sh
Traceback (most recent call last):
  File "/apps/spack/spack-0.15.0/bin/spack", line 64, in <module>
    sys.exit(spack.main.main())
  File "/apps/spack/spack-0.15.0/lib/spack/spack/main.py", line 715, in main
    print_setup_info(*args.print_shell_vars.split(','))
  File "/apps/spack/spack-0.15.0/lib/spack/spack/main.py", line 629, in print_setup_info
    shell_set('_sp_sys_type', spack.architecture.sys_type())
  File "/apps/spack/spack-0.15.0/lib/spack/llnl/util/lang.py", line 178, in _memoized_function
    func.cache[args] = func(*args)
  File "/apps/spack/spack-0.15.0/lib/spack/spack/architecture.py", line 533, in sys_type
    arch = Arch(platform(), 'default_os', 'default_target')
  File "/apps/spack/spack-0.15.0/lib/spack/llnl/util/lang.py", line 178, in _memoized_function
    func.cache[args] = func(*args)
  File "/apps/spack/spack-0.15.0/lib/spack/spack/architecture.py", line 517, in platform
    return platform_cls()
  File "/apps/spack/spack-0.15.0/lib/spack/spack/platforms/cray.py", line 60, in __init__
    raise NoPlatformError()
_**spack.architecture.NoPlatformError: Could not determine a platform for this machine.**_


**on v0.14**, I could at least unload craype and initialize and recognize the arch:
ln01-nmn:/apps/spack/spack-0.14.2/share/spack # . ./setup-env.sh
ln01-nmn:/apps/spack/spack-0.14.2/share/spack # spack arch
linux-sles15-zen2

Thanks,
Chris

We need more documentation for standup of Cray systems with release 15.  I had to hack around a bit but finally resolved that I needed to set SPACK_FRONT_END and SPACK_BACK_END only to hit a different issue.  In operating_systems/cray_backend.py line 100 specifies the "RELEASE" from the cle-release file.  This value does not exist in the cle-release file.  By the way, on Shasta, I had to set SPACK_FRONT_END=zen and SPACK_BACK_END=zen.  

PRODUCT="Cray's Linux Environment"
OS=SLES15SP1
ARCH=x86_64
VERSION=1.2.0
DATE=20200421195827

So I changed operating_systems/cray_backend.py from RELEASE to VERSION and at least now spack loads (spack list doesn't crash).  Next to see if anything can build with Cray CCE and cray-mpich (which is the real test)
Are there other portions of configuration I'm missing here?  I've been able to get intel and gcc to build (via direct calls to their executables (not via the PrgEnv-intel or PrgEnv-gcc wrappers through Cray's cc/CC/ftn).

Donald.Duck@ln01-nmn:~/spack-develop/share/spack> spack install zlib %cce
self._avail_targets -> zen
SPACK_BACK_END -> zen
self._avail_targets -> zen
SPACK_BACK_END -> zen
==> 13616: Installing zlib
==> Using cached archive: /lfs/h1/home/Donald.Duck/spack-develop/var/spack/cache/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
==> Staging archive: /tmp/Donald.Duck/spack-stage/spack-stage-zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2/zlib-1.2.11.tar.gz
==> Created stage in /tmp/Donald.Duck/spack-stage/spack-stage-zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2
==> Applied patch /lfs/h1/home/Donald.Duck/spack-develop/var/spack/repos/builtin/packages/zlib/w_patch.patch
==> 13616: zlib: Building zlib [Package]
==> 13616: zlib: Executing phase: 'install'
==> Error: ProcessError: Command exited with status 1:
    './configure' '--prefix=/lfs/h1/home/Donald.Duck/spack-develop/opt/spack/cray-cnl1-zen/cce-10.0.1/zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2'
See build log for details:
  /tmp/Donald.Duck/spack-stage/spack-stage-zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2/spack-build-out.txt

==> Error: Failed to install zlib due to ChildError: ProcessError: Command exited with status 1:
    './configure' '--prefix=/lfs/h1/home/Donald.Duck/spack-develop/opt/spack/cray-cnl1-zen/cce-10.0.1/zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2'See build log for details:
  /tmp/Donald.Duck/spack-stage/spack-stage-zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2/spack-build-out.txt
Traceback (most recent call last):
  File "/lfs/h1/home/Donald.Duck/spack-develop/lib/spack/spack/build_environment.py", line 838, in child_process
    return_value = function()
  File "/lfs/h1/home/Donald.Duck/spack-develop/lib/spack/spack/installer.py", line 1140, in build_process
    phase(pkg.spec, pkg.prefix)
  File "/lfs/h1/home/Donald.Duck/spack-develop/lib/spack/spack/package.py", line 108, in phase_wrapper
    phase(spec, prefix)
  File "/lfs/h1/home/Donald.Duck/spack-develop/var/spack/repos/builtin/packages/zlib/package.py", line 52, in install
  File "/lfs/h1/home/Donald.Duck/spack-develop/lib/spack/spack/util/executable.py", line 189, in __call__
    proc.returncode, long_msg)
spack.util.executable.ProcessError: Command exited with status 1:
    './configure' '--prefix=/lfs/h1/home/Donald.Duck/spack-develop/opt/spack/cray-cnl1-zen/cce-10.0.1/zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2'

Donald.Duck@ln01-nmn:~/spack-develop/share/spack> less -X /tmp/Donald.Duck/spack-stage/spack-stage-zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2/spack-build-out.txt
==> 13616: zlib: Executing phase: 'install'
==> [2020-07-07-18:27:15.235483, 13638] './configure' '--prefix=/lfs/h1/home/Donald.Duck/spack-develop/opt/spack/cray-cnl1-zen/cce-10.0.1/zlib-1.2.11-rybkcb7gkroaij73khbg2band6dviiq2'
**Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).**
** ./configure aborting.






@becker33 is this still a problem?
I don't know. I haven't looked at this since the v0.15.0 release.

@riscy768 do you know whether the operating system is the same on the FE and BE nodes on the systems you're discussing? I know Cray has been making some changes to their programming environment, but I don't have access to a Shasta system to debug on.

Resolving patch:
diff --git a/lib/spack/spack/operating_systems/cray_backend.py b/lib/spack/spack/operating_systems/cray_backend.py
--- a/lib/spack/spack/operating_systems/cray_backend.py
+++ b/lib/spack/spack/operating_systems/cray_backend.py
@@ -97,6 +97,9 @@ def __str__(self):
     def _detect_crayos_version(cls):
         if os.path.isfile(_cle_release_file):
             release_attrs = read_cle_release_file()
+            if 'RELEASE' not in release_attrs:
+                # This Cray system uses a base OS not CLE/CNL
+                return None
             v = spack.version.Version(release_attrs['RELEASE'])
             return v[0]
         elif os.path.isfile(_clerelease_file):
diff --git a/lib/spack/spack/platforms/cray.py b/lib/spack/spack/platforms/cray.py
--- a/lib/spack/spack/platforms/cray.py
+++ b/lib/spack/spack/platforms/cray.py
@@ -20,7 +20,7 @@
 _craype_name_to_target_name = {
     'x86-cascadelake': 'cascadelake',
     'x86-naples': 'zen',
-    'x86-rome': 'zen',  # Cheating because we have the wrong modules on rzcrayz
+    'x86-rome': 'zen2',
     'x86-skylake': 'skylake_avx512',
     'mic-knl': 'mic_knl',
     'interlagos': 'bulldozer',


Test patch: