Repository: spack/spack

Issue:
Prepare hpcviewer for Mac OS platform
 * Adapt URL parsing for Mac OS
 * Adapt install method for Mac OS

Todo:
- [x] Update versions
- [x] Polish implementation

cc: @mwkrentel 

Note: We need #23054 for testing this PR:

```console
→ spack spec -I hpcviewer
Input spec
--------------------------------
 -   hpcviewer

Concretized
--------------------------------
 -   hpcviewer@2021.03%apple-clang@12.0.0 arch=darwin-catalina-skylake
 -       ^openjdk@11.0.2%apple-clang@12.0.0 arch=darwin-catalina-skylake

→ spack install hpcviewer
==> Installing openjdk-11.0.2-kvhanirclvrqbpqlytiiosgbycxynphf
==> No binary for openjdk-11.0.2-kvhanirclvrqbpqlytiiosgbycxynphf found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check apple-clang@12.0.0 for further details]
==> Fetching https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_osx-x64_bin.tar.gz
...
[+] /Users/pramod/Downloads/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/openjdk-11.0.2-kvhanirclvrqbpqlytiiosgbycxynphf
==> Installing hpcviewer-2021.03-cz4t4lloaq7yig52b6obbyknpc4rgyjf
==> No binary for hpcviewer-2021.03-cz4t4lloaq7yig52b6obbyknpc4rgyjf found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check apple-clang@12.0.0 for further details]
==> Fetching http://hpctoolkit.org/download/hpcviewer/2021.03/hpcviewer-2021.03-macosx.cocoa.x86_64.zip
....
==> hpcviewer: Successfully installed hpcviewer-2021.03-cz4t4lloaq7yig52b6obbyknpc4rgyjf
  Fetch: 5.79s.  Build: 0.93s.  Total: 6.71s.
[+] /Users/pramod/Downloads/spack/opt/spack/darwin-catalina-skylake/apple-clang-12.0.0/hpcviewer-2021.03-cz4t4lloaq7yig52b6obbyknpc4rgyjf
```

and it should be ready:

```
→ module load hpcviewer-2021.03-apple-clang-12.0.0-cz4t4ll
→ hpcviewer
```
![image](https://user-images.githubusercontent.com/666852/115094017-622dc700-9f1c-11eb-8894-6fccce376f63.png)



Discussion:
@pramodk Thanks for doing this !

I checked the Linux side, and that seems Ok.  I guess if it was me,
I'd make the new keys a triple, eg, `('2021.03', 'linux', 'x86_64')`
instead of hyphenated, but whatever.

Also, let's update the package description to:

```
class Hpcviewer(Package):
    """Binary distribution of hpcviewer and integrated traceviewer
    for the Rice HPCToolkit (MacOS x86_64, and Linux x86_64, ppc64le
    and aarch64).  Note: hpctoolkit databases are platform independent,
    so you don't need to install hpctoolkit to run the viewer, and
    it's common to run hpcrun and hpcviewer on different machines."""
```

As for the Macos versions, my Macbook is currently stuck in a bad
state where it won't run the Xcode tools (git).  When I get that
resolved, then I'll check out the Macos side, which is the whole point
of the PR.

But without running this, I guess I don't see how spack connects the
Java 11 dependency for hpcviewer on a Mac that doesn't already have
Java 11.

@laksono Any thoughts ?
> I checked the Linux side, and that seems Ok. I guess if it was me,
> I'd make the new keys a triple, eg, ('2021.03', 'linux', 'x86_64')
> instead of hyphenated, but whatever.

I was thinking that but I did this way because `openjdk` package has this scheme. So thought to do the same way.

>  When I get that resolved, then I'll check out the Macos side, which is the whole point of the PR.

Ok. I am testing on Mac OS. So I am sure that this is working.

> But without running this, I guess I don't see how spack connects the Java 11 dependency for hpcviewer on a Mac that doesn't already have Java 11.

hpcviewer has dependency with Java. So it installs OpenJDK and rest should work? (May be I should remove my system Java and test again).


Or, you could install system Java 8 and see how the viewer reacts.
Or, install both old (java 8) and new (java 11) versions of the viewer and 
see if they both run without making any system adjustment.
But I never run spack on Macos, so I'm just guessing.
I did following:

```bash
# remove java
sudo rm -rf /Library/Internet\ Plug-Ins/JavaAppletPlugin.plugin  
sudo rm -rf /Library/PreferencePanes/JavaControlPanel.prefPane  
sudo rm -rf /Library/Application\ Support/Oracle/Java/ 
sudo rm -rf /Library/Java/JavaVirtualMachines 

# now there is no java
○ → java
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

# install fresh hpcviewer with openjdk

○ → spack install hpcviewer
==> Installing openjdk-11.0.2-pdv6wir3l7vnqmmz6mrf2fnnlcv6fbf4
==> No binary for openjdk-11.0.2-pdv6wir3l7vnqmmz6mrf2fnnlcv6fbf4 found: installing from source
/Users/pramod/Downloads/spack/lib/spack/spack/architecture.py:205: UserWarning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check apple-clang@12.0.0 for further details]
  warnings.warn(msg.format(compiler))
==> Fetching https://download.java.net/java/GA/jdk11/9/GPL/openjdk-11.0.2_osx-x64_bin.tar.gz
########################################################################################################################################################################## 100.0%
==> No patches needed for openjdk
==> openjdk: Executing phase: 'install'
==> openjdk: Successfully installed openjdk-11.0.2-pdv6wir3l7vnqmmz6mrf2fnnlcv6fbf4
  Fetch: 12.60s.  Build: 3.78s.  Total: 16.38s.
[+] /Users/pramod/Downloads/spack/opt/spack/darwin-bigsur-skylake/apple-clang-12.0.0/openjdk-11.0.2-pdv6wir3l7vnqmmz6mrf2fnnlcv6fbf4
==> Installing hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5
==> No binary for hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5 found: installing from source
/Users/pramod/Downloads/spack/lib/spack/spack/architecture.py:205: UserWarning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check apple-clang@12.0.0 for further details]
  warnings.warn(msg.format(compiler))
==> Using cached archive: /Users/pramod/Downloads/spack/var/spack/cache/_source-cache/archive/ef/ef7598b2a8d6574f315915452015d4b50ad8631165a872d1fbccac80284db160.zip
==> No patches needed for hpcviewer
==> hpcviewer: Executing phase: 'install'
==> hpcviewer: Successfully installed hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5
  Fetch: 0.09s.  Build: 0.92s.  Total: 1.01s.
[+] /Users/pramod/Downloads/spack/opt/spack/darwin-bigsur-skylake/apple-clang-12.0.0/hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5

# try only hpcviewer module

○ → module show hpcviewer-2021.03-apple-clang-12.0.0-rsjhuiq
-------------------------------------------------------------------
/Users/pramod/Downloads/spack/share/spack/modules/darwin-bigsur-skylake/hpcviewer-2021.03-apple-clang-12.0.0-rsjhuiq:

module-whatis   {Binary distribution of hpcviewer and hpctraceviewer for the Rice HPCToolkit (Linux x86_64, ppc64le and aarch64). Note: hpctoolkit databases are platform independent, so you don't need to install hpctoolkit to run the viewers and it's common to run hpcrun and hpcviewer on different machines.}
prepend-path    PATH /Users/pramod/Downloads/spack/opt/spack/darwin-bigsur-skylake/apple-clang-12.0.0/hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5/bin
prepend-path    CMAKE_PREFIX_PATH /Users/pramod/Downloads/spack/opt/spack/darwin-bigsur-skylake/apple-clang-12.0.0/hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5/
-------------------------------------------------------------------

○ → module load hpcviewer-2021.03-apple-clang-12.0.0-rsjhuiq
○ → which hpcviewer
/Users/pramod/Downloads/spack/opt/spack/darwin-bigsur-skylake/apple-clang-12.0.0/hpcviewer-2021.03-rsjhuiqdi2wp3sn2a4yeydwrsoluiht5/bin/hpcviewer
○ → hpcviewer
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

Script started, output file is /dev/null
java_home: unrecognized option `--request'
The operation couldn’t be completed. Unable to locate a Java Runtime.
Please visit http://www.java.com for information on installing Java.

# there is java module auto-loaded yet and hence hpcviewer didn't work. Try loading it:

○ → module load openjdk-11.0.2-apple-clang-12.0.0-pdv6wir
○ → hpcviewer
```

![image](https://user-images.githubusercontent.com/666852/115140966-41f42a00-a03a-11eb-916c-368728bf81a8.png)

So everything looking good!
@pramodk I get different results and I think I see why.  For me, java
dies with "does not contain the JNI_Create_JavaVM symbol." because
I don't have java 11 installed.

First, you're using modules, but I don't have modules installed.
Did you do this with spack's bootstrap for modules?
Or, are the modules somehow builtin to the Macos shell?

Also, it looks like you load both the viewer and java modules, is that
right?  That's how hpcviewer finds java by loading the java module.

I'd like to see if there is a way to make this work without using
modules, or if we have to use modules, then write the java path into
the hpcviewer module.  On Linux, we do this with the install script.

I think we're close, but I'd like to make this a little easier for the
user and maybe more macos-thematic, if that's possible.
> First, you're using modules, but I don't have modules installed.
> Did you do this with spack's bootstrap for modules?
> Or, are the modules somehow builtin to the Macos shell?

I typically use `brew` to install common packages:

```
brew install modules
source /usr/local/opt/modules/init/bash
```
and it should setup `module`.

> Also, it looks like you load both the viewer and java modules, is that
> right? That's how hpcviewer finds java by loading the java module.

Thats correct. But note that we don't need modules nowadays i.e. you can justo use `spack load` to load packages: `spack load openjdk hpcviewer` 

> I'd like to see if there is a way to make this work without using
> modules, or if we have to use modules, then write the java path into
> the hpcviewer module.

I was thinking the same i.e. I would also prefer not generate module or load openjdk. So would support adding env variables in hpcviewer itself.
Ok, I think `spack load` is a good alternative to modules.

I fixed my Xcode problems, but I'm seeing some strange (and broken)
results depending on how I run things.  Let me check this with
@laksono for the best way to launch the viewer and connect with java.
Ok, I discussed the Macos side with Laksono.  There are a few more
things we need to do to make the Macos side work like it should.
They're all pretty short but important.

The first three are for the darwin install only.  The last one is for
the linux install.

(1) There needs to be an extra directory between the spack install
prefix and the Contents directory, named 'hpcviewer.app'.  So:

```
  ...
    darwin-bigsur-broadwell
      apple-clang-12.0.0
        hpcviewer-2021.03-gpqletix66isc3fr53kfgq5rl5teqpt2
	  hpcviewer.app
	    Contents
	      Eclipse
              Info.plist
              MacOS
              Resources
```

The hpcviewer.app directory is very important for the Macos side.  You
can either cd to the `hpcviewer-2021.03` directory and say `open hpcviewer.app`,
or you can click your way through Finder and double
click on that icon.  Either of those will open the viewer.

(2) I would prefer making the `bin` symlink relative, so it's contents
would be `hpcviewer.app/Contents/MacOS`.

(3) We need to add two lines to the beginning of the `hpcviewer.ini`
file in the `Eclipse` directory.  The lines are: `-vm` and the full
path to the `java` binary in the `bin` directory of the java/jdk
package.  These go on separate lines as the first two lines of the
file.  For example:

```
hpcviewer.ini:

-vm
/Users/krentel/viewer/install/darwin-bigsur-broadwell/apple-clang-12.0.0/openjdk-11.0.2-d5g4zxhilgfjuuugcvjhywkxe2ybxbtu/bin/java
-startup
../Eclipse/plugins/org.eclipse.equinox.launcher_1.6.0.v20200915-1508.jar
...
```

Note that the value for `-vm` (line 2) ends with `.../bin/java`  (it's a long line).
Also, it's just one dash in `-vm`.

This is very important for how the viewer finds the right java.  It
also means that we don't really need to add a prepend-path PATH or
JAVA_HOME for the viewer module to find java.


(4) The `@when('platform=linux')` decorator is not correct because it
misses cray.  Note that in this context, `platform` is the spack
platform (where linux and cray are different), not the python
platform.  To fix it, we need to use the python plaform like this:

```
    # Both hpcviewer and trace viewer have an install script.
    @when(platform.system().lower() == 'linux')
    def install(self, spec, prefix):
        ...
```

So, it seems like a lot, but they're all short and all necessary.  In
the end, it will be thematically correct for Macos.

Btw, I figured out how hpcviewer was picking up the correct java
without your doing anything explicit for it.  In spack, hpcviewer
depends on openjdk, so when you ran `spack load hpcviewer`, spack
automatically also loaded the openjdk package.  You can see this with
`spack find --loaded`.

Or, with modules, if you load both, then you'll get the right java.

Anyway, make these changes, then I'll run through the steps to test it
one last time, and then we'll be done!

Thanks !



Resolving patch:
diff --git a/var/spack/repos/builtin/packages/hpcviewer/package.py b/var/spack/repos/builtin/packages/hpcviewer/package.py
--- a/var/spack/repos/builtin/packages/hpcviewer/package.py
+++ b/var/spack/repos/builtin/packages/hpcviewer/package.py
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 
+import os
 import os.path
 import platform
 
@@ -28,17 +29,35 @@ def trace_url(ver, mach):
     ).format(ver, ver2, mach)
 
 
+def darwin_url(ver, mach):
+    return (
+        'http://hpctoolkit.org/download/hpcviewer/{0}/'
+        'hpcviewer-{0}-macosx.cocoa.{1}.zip'
+    ).format(ver, mach)
+
+
 class Hpcviewer(Package):
-    """Binary distribution of hpcviewer and hpctraceviewer for the Rice
-    HPCToolkit (Linux x86_64, ppc64le and aarch64).  Note: hpctoolkit
-    databases are platform independent, so you don't need to install
-    hpctoolkit to run the viewers and it's common to run hpcrun and
-    hpcviewer on different machines."""
+    """Binary distribution of hpcviewer and integrated hpctraceviewer for
+    the Rice HPCToolkit (Linux x86_64, ppc64le and aarch64, and MacOSX
+    x86_64).
+
+    Note: hpctoolkit databases are platform independent, so you don't
+    need to install hpctoolkit to run the viewers and it's common to
+    run hpcrun and hpcviewer on different machines.
+    """
 
     homepage = "http://hpctoolkit.org"
     maintainers = ['mwkrentel']
 
+    darwin_sha = {
+        ('2021.10', 'x86_64'):  '0b71f2d63d99eb00fbaf9c03cf8632c198627c80e4372eeec5f20864509cbbe8',
+        ('2021.05', 'x86_64'):  '4643567b41dddbbf9272cb56b0720f4eddfb144ca05aaad7d08c878ffaf8f2fa',
+    }
+
     viewer_sha = {
+        ('2021.10', 'aarch64'): 'c696a0ecc6696f9979de3940b5471a3d99c8d573736cabb24b86255d860a23dc',
+        ('2021.10', 'ppc64le'): 'f0eda3510b71fd9115c5653efba29aaefcb335c66b118cf63f36e1353c39e24e',
+        ('2021.10', 'x86_64'):  'd5a444e28d6c9d1a087c39bd3ffe55c6f982dc37a7a743b83bbba2fbfc7ca7c6',
         ('2021.05', 'aarch64'): 'a500bf14be14ca9b08a8382f1d122f59b45690b6a567df0932fc2cabd6382a9a',
         ('2021.05', 'ppc64le'): 'd39f9f6556abcd5a184db242711b72b2e8571d0b78bb08d0e497fd4e6dbe87a1',
         ('2021.05', 'x86_64'):  'f316c1fd0b134c96392cd4eb5e5aa2bffa36bd449f401d8fe950ab4f761c34ab',
@@ -121,25 +140,66 @@ class Hpcviewer(Package):
         ('2019.02', 'ppc64le'): '01a159306e7810efe07157ec823ac6ca7570ec2014c95db599a3f90eee33355c',
     }
 
-    for key in viewer_sha.keys():
-        if key[1] == platform.machine():
-            version(key[0], url=viewer_url(*key), sha256=viewer_sha[key])
-
-            # Current versions include the viewer and trace viewer in
-            # one tar file.  Before 2020.07, the trace viewer was a
-            # separate tar file (resource).
-            if key in trace_sha:
-                resource(name='hpctraceviewer', url=trace_url(*key),
-                         sha256=trace_sha[key], placement='TRACE',
-                         when='@{0}'.format(key[0]))
+    system = platform.system().lower()
+    machine = platform.machine().lower()
+
+    # Versions for MacOSX / Darwin
+    if system == 'darwin':
+        for key in darwin_sha.keys():
+            if key[1] == machine:
+                version(key[0], url=darwin_url(*key), sha256=darwin_sha[key])
+
+    # Versions for Linux and Cray front-end
+    if system == 'linux':
+        for key in viewer_sha.keys():
+            if key[1] == machine:
+                version(key[0], url=viewer_url(*key), sha256=viewer_sha[key],
+                        deprecated=(key[0] <= '2020.01'))
+
+                # Current versions include the viewer and trace viewer in
+                # one tar file.  Before 2020.07, the trace viewer was a
+                # separate tar file (resource).
+                if key in trace_sha:
+                    resource(name='hpctraceviewer', url=trace_url(*key),
+                             sha256=trace_sha[key], placement='TRACE',
+                             when='@{0}'.format(key[0]))
 
     depends_on('java@11:', type=('build', 'run'), when='@2021.0:')
     depends_on('java@8', type=('build', 'run'), when='@:2020')
 
-    conflicts('platform=darwin', msg='hpcviewer requires a manual install on MacOS, see homepage')
+    # Install for MacOSX / Darwin
+    @when('platform=darwin')
+    def install(self, spec, prefix):
+        # Add path to java binary to hpcviewer.ini file.
+        ini_file = join_path('Contents', 'Eclipse', 'hpcviewer.ini')
+        java_binary = join_path(spec['java'].prefix.bin, 'java')
+        filter_file('(-startup)', '-vm\n' + java_binary + '\n' + r'\1',
+                    ini_file, backup=False)
+
+        # Copy files into prefix/hpcviewer.app.
+        app_dir = join_path(prefix, 'hpcviewer.app')
+        mkdirp(app_dir)
+        install_tree('.', app_dir)
+
+        # Add launch script to call 'open' on app directory.
+        mkdirp(prefix.bin)
+        viewer_file = join_path(prefix.bin, 'hpcviewer')
+        with open(viewer_file, 'w') as file:
+            file.write('#!/bin/sh\n')
+            file.write('open ' + app_dir + '\n')
+        os.chmod(viewer_file, 0o755)
+
+    # Install for Cray front-end is the same as Linux.
+    @when('platform=cray')
+    def install(self, spec, prefix):
+        self.linux_install(spec, prefix)
+
+    @when('platform=linux')
+    def install(self, spec, prefix):
+        self.linux_install(spec, prefix)
 
     # Both hpcviewer and trace viewer have an install script.
-    def install(self, spec, prefix):
+    def linux_install(self, spec, prefix):
         args = [
             '--java', spec['java'].home,
             prefix


Test patch: