Repository: spack/spack

Issue:
Installation issue: py-astropy
Hi all,
Sorry to bother you, but I was wondering if any other Mac user had encountered a similar issue before.
I tried to install the package through the commands:
```console
spack install py-astropy
```
and
```console
spack install py-astropy ^wcslib
```
But got an error message during the build stage, due to missing (or not linked) header files.
Such as:
```bash
 >> 1229    astropy/wcs/include/astropy_wcs/util.h:21:10: fatal error: 'wcs.h' file not found
```
That can be seen in the output file: [spack-build-out.txt](https://github.com/spack/spack/files/4131212/spack-build-out.txt).

Do you have any suggestions on how I can try to fix this issue or what I am doing wrong ?

Thank you

### Platform and user environment

Mac OS Catalina:
```commandline
Darwin LDianaAmorim-M30.dhcp.lbnl.us 19.2.0 Darwin Kernel Version 19.2.0: Sat Nov  9 03:47:04 PST 2019; root:xnu-6153.61.1~20/RELEASE_X86_64 x86_64
OS version: 10.15.2
Clang version: 11.0.0 x86_64-apple-darwin19.2.0
spack git log: commit 7a0990903fbca221e044eb43edae6fd39dcca4eb
python version: Python 3.7.6 (installed and loaded via spack)
``` 

.spack/darwin/packages.yaml:
```cpp
packages:
  all:
    providers:
      mpi: [mpich, openmpi]
```
.spack/darwin/compilers.yaml:
```cpp
compilers:
- compiler:
    spec: clang@11.0.0-apple
    paths:
      cc: /usr/bin/clang
      cxx: /usr/bin/clang++
      f77: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/gfortran
      fc: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/gfortran
    flags: {}
    operating_system: catalina
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
- compiler:
    spec: gcc@9.2.0
    paths:
      cc: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/gcc
      cxx: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/g++
      f77: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/gfortran
      fc: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.0-apple/gcc-9.2.0-exw25ccpcwqlkcvuwn266kvwqzxbyelp/bin/gfortran
    flags: {}
    operating_system: catalina
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
```


### Additional information

```console
$ spack spec --install-status
```
led to the file:
[output.txt](https://github.com/spack/spack/files/4131230/output.txt)

Rebuilding the package with the following options:
```console
$ spack -d install -j 1 <spec>
```
led to the files:

[spack-cc-py-astropy-6ggjsuu.in.log](https://github.com/spack/spack/files/4131235/spack-cc-py-astropy-6ggjsuu.in.log)
[spack-cc-py-astropy-6ggjsuu.out.log](https://github.com/spack/spack/files/4131236/spack-cc-py-astropy-6ggjsuu.out.log)



Discussion:
Any updates on this issue?
Are you having a similar issue? Does `wcs.h` appear anywhere in the `wcslib` prefix (I think that's the dependency that would provide it)? If so can you include where you see it? If not, it's possible that `py-astropy` needs another dependency.
It does not...not that I could find.  I can see that it tries to build with a flag ``--use-system-wcslib`` So somehow it might be that it thinks I have it somewhere, but I can't find it on my system anywhere.
Hi, sorry. I haven't tested this in a long while.
Hi thanks, I'm stuck on this and use any thoughts.
It seems that you still need to have the `wcslib` package installed, from looking at:
https://github.com/spack/spack/blob/ec1237479ee8786c808db722ecb0378136cd22d3/var/spack/repos/builtin/packages/py-astropy/package.py#L58

Can you do `spack install wcslib` and then `spack load wcslib`?
Or do you get errors?

Can you try commenting out that line and the one:
https://github.com/spack/spack/blob/ec1237479ee8786c808db722ecb0378136cd22d3/var/spack/repos/builtin/packages/py-astropy/package.py#L66
just for curiosity?

That didn't fix the issue (exact same error messages), but I am wondering if there is a way to "make clean" for py-astropy and start again.  Searching the web for how to do this now... any pointers/ideas would be appreciated

==> Error: ProcessError: Command exited with status 1:
    '/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.7-cj6tnvnjdoemms4chr3kvlkdix7vrhkd/bin/python3.7' '-s' 'setup.py' '--no-user-cfg' 'build' '--use-system-libraries' '--use-system-erfa' '--use-system-cfitsio' '--use-system-expat' '-j' '8'

3 errors found in build log:
     1237    /Users/marc/src/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
              -I/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.7-cj6tnvnjdoemms4chr3kvlkdix7vrhkd/include/python3
             .7m -c astropy/utils/src/compiler.c -o build/temp.macosx-10.14.6-x86_64-3.7/astropy/utils/src/compiler.o
     1238    /Users/marc/src/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
              -I/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/py-numpy-1.19.0-3wlusfwdmawofdunv3u4awifti3hadbg/lib/python3.
             7/site-packages/numpy/core/include -I/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.7-cj6tnvnjdoemms
             4chr3kvlkdix7vrhkd/include/python3.7m -c astropy/convolution/src/convolve.c -o build/temp.macosx-10.14.6-x86_64-3.7/astropy/convolution/
             src/convolve.o -UNDEBUG -fPIC
     1239    /Users/marc/src/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
              -DECHO -DWCSTRIG_MACRO -DASTROPY_WCS_BUILD -D_GNU_SOURCE -DNDEBUG -UDEBUG -I/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-
             10.0.1-apple/py-numpy-1.19.0-3wlusfwdmawofdunv3u4awifti3hadbg/lib/python3.7/site-packages/numpy/core/include -Iastropy/modeling/src -I/U
             sers/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.7-cj6tnvnjdoemms4chr3kvlkdix7vrhkd/include/python3.7m -
             c astropy/modeling/src/projections.c -o build/temp.macosx-10.14.6-x86_64-3.7/astropy/modeling/src/projections.o -Wno-strict-prototypes -
             Wno-unused-function -Wno-unused-value -Wno-uninitialized
     1240    /Users/marc/src/spack/lib/spack/env/clang/clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -fPIC
              -DECHO -DWCSTRIG_MACRO -DASTROPY_WCS_BUILD -D_GNU_SOURCE -DNDEBUG -UDEBUG -I/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-
             10.0.1-apple/py-numpy-1.19.0-3wlusfwdmawofdunv3u4awifti3hadbg/lib/python3.7/site-packages/numpy/core/include -Iastropy/wcs/include -I/Us
             ers/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/python-3.7.7-cj6tnvnjdoemms4chr3kvlkdix7vrhkd/include/python3.7m -c
              astropy/wcs/src/distortion.c -o build/temp.macosx-10.14.6-x86_64-3.7/astropy/wcs/src/distortion.o -Wno-strict-prototypes -Wno-unused-fu
             nction -Wno-unused-value -Wno-uninitialized
     1241    In file included from astropy/wcs/src/distortion.c:6:
     1242    In file included from astropy/wcs/include/astropy_wcs/distortion.h:9:
  >> 1243    astropy/wcs/include/astropy_wcs/util.h:21:10: fatal error: 'wcs.h' file not found

After you load the `wcslib` module, do you know where: `wcs.h` is?

If it is in your `spack` folder, then can you try adding the wcslib module as a (fake) external package in your `packages.yaml`?
I did that to use `cmake` with gcc using its `spack` path:
```
packages:
  cmake:
    buildable: False
    paths:
      cmake@3.17.0: /Users/LDianaAmorim/Documents/opt/spack/opt/spack/darwin-catalina-x86_64/clang-11.0.3-apple/cmake-3.17.0-6exaq5d4lp7sxf77nhk6kfkuajiy3qwm
```
So you can try the same but to your `wcslib` path.
> I am wondering if there is a way to "make clean" for py-astropy and start again.

If `spack install` fails for some package and you try again, Spack will automatically clean everything and start from the beginning with the package that failed.

Spack provides a means to "debug" an installation with the `spack build-env` command, which will set up the environment appropriately. See also: https://spack.readthedocs.io/en/latest/packaging_guide.html#spack-build-env


@LDAmorim I see a wcs.h file at

/Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/wcslib-6.4-rxqn53e3azxkvkxoe7qxy7nb2aga273g/include/wcslib/wcs.h

linked from /Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/wcslib-6.4-rxqn53e3azxkvkxoe7qxy7nb2aga273g/include/wcslib-6.4/

I'll try modifying the ``packages.yaml`` file as you suggest
> I see a wcs.h file at
> /Users/marc/src/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.1-apple/wcslib-6.4-rxqn53e3azxkvkxoe7qxy7nb2aga273g/include/wcslib/wcs.h

One possible reason `py-astropy` cannot find these headers is that they don't appear in the default location that Spack expects: that would be `<wcslib-prefix>/include/wcs.h`.

You can define a `headers` function in the `wcslib` package to tell Spack where to find the headers in the `wcslib` prefix (in `<wcslib-prefix>/include/wcslib/`). `openmpi` has an example of that function.
I could try copying/modifying the headers function from openmpi, but your observation suggests a bigger problem with my install.  Can you suggest how ``spack install wcslib`` has put this lib in a place that is not the "default location"?  
Indeed he has them in `include/wcslib/wcs.h` instead of `include/wcs.h`.
Ha!  Manually linking the includes to the expected path allowed the install to complete.  This may be an error in the wcslib spack install?  Anyway, awesome...thank you both so much!

> I could try copying/modifying the headers function from openmpi, but your observation suggests a bigger problem with my install. Can you suggest how spack install wcslib has put this lib in a place that is not the "default location"?

When I look at the Spack `wcslib` package I don't see any configuration related to the headers directory, so I think this is the location chosen by `wcslib`. Spack generally doesn't adjust the layout of the package installation other than to try to set the root prefix of the installation.

So there's two ways to address the issue:

* Make `wcslib` install the headers to `include/` (instead of `include/wcslib/`)
* Allow dependent packages to find headers in `include/wcslib` (i.e. https://github.com/spack/spack/issues/14679#issuecomment-662022982)
You're very welcome! I'm happy it worked, but it is just a work around as @scheibelp pointed out there is an actual solution.

I can try to work on the headers PR. But I never did it before.
Would it be ok for me to ask you both to help me review it?
Sure, I'd like to learn about how all this works as well.
> Would it be ok for me to ask you both to help me review it?

Yes! I'd really appreciate it if you submitted a PR and would be happy to review it.
Hi, I tried to make some changes in https://github.com/spack/spack/pull/17666 but haven't been successful so far.
Can you point me to other files that managed to do the headers path change correctly?
Thanks,
Diana
I think the correct solution is to add a build dependency on pkg-config. I'll try this out and put together a PR soon.

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-astropy/package.py b/var/spack/repos/builtin/packages/py-astropy/package.py
--- a/var/spack/repos/builtin/packages/py-astropy/package.py
+++ b/var/spack/repos/builtin/packages/py-astropy/package.py
@@ -12,9 +12,12 @@ class PyAstropy(PythonPackage):
     package for Astronomy in Python and foster interoperability between
     Python astronomy packages."""
 
-    homepage = 'http://www.astropy.org/'
-    url = 'https://pypi.io/packages/source/a/astropy/astropy-3.2.1.tar.gz'
+    homepage = 'https://astropy.org/'
+    url = 'https://pypi.io/packages/source/a/astropy/astropy-4.0.1.post1.tar.gz'
 
+    install_time_test_callbacks = ['install_test', 'import_module_test']
+
+    version('4.0.1.post1', sha256='5c304a6c1845ca426e7bc319412b0363fccb4928cb4ba59298acd1918eec44b5')
     version('3.2.1', sha256='706c0457789c78285e5464a5a336f5f0b058d646d60f4e5f5ba1f7d5bf424b28')
     version('2.0.14', sha256='618807068609a4d8aeb403a07624e9984f566adc0dc0f5d6b477c3658f31aeb6')
     version('1.1.2', sha256='6f0d84cd7dfb304bb437dda666406a1d42208c16204043bc920308ff8ffdfad1')
@@ -29,14 +32,17 @@ class PyAstropy(PythonPackage):
     depends_on('python@2.7:2.8,3.3:', when='@1.2:', type=('build', 'run'))
     depends_on('python@2.6:', type=('build', 'run'))
     depends_on('py-setuptools', type='build')
+    depends_on('py-cython@0.29.13:', type='build')
+    depends_on('py-numpy@1.16:', when='@4.0:', type=('build', 'run'))
     depends_on('py-numpy@1.13:', when='@3.1:', type=('build', 'run'))
     depends_on('py-numpy@1.10:', when='@3.0:', type=('build', 'run'))
     depends_on('py-numpy@1.9:', when='@2.0:', type=('build', 'run'))
     depends_on('py-numpy@1.7:', when='@1.2:', type=('build', 'run'))
     depends_on('py-numpy', type=('build', 'run'))
+    depends_on('pkgconfig', type='build')
 
     # Optional dependencies
-    depends_on('py-scipy', when='+extras', type=('build', 'run'))
+    depends_on('py-scipy@0.18:', when='+extras', type=('build', 'run'))
     depends_on('py-h5py', when='+extras', type=('build', 'run'))
     depends_on('py-beautifulsoup4', when='+extras', type=('build', 'run'))
     depends_on('py-html5lib', when='+extras', type=('build', 'run'))
@@ -53,7 +59,6 @@ class PyAstropy(PythonPackage):
     depends_on('py-asdf@2.3:', when='+extras', type=('build', 'run'))
     depends_on('py-bottleneck', when='+extras', type=('build', 'run'))
     depends_on('py-pytest', when='+extras', type=('build', 'run'))
-    depends_on('py-cython', type='build')
 
     # System dependencies
     depends_on('erfa')
@@ -81,3 +86,7 @@ def build_args(self, spec, prefix):
             args.extend(['-j', str(make_jobs)])
 
         return args
+
+    def install_test(self):
+        with working_dir('spack-test', create=True):
+            python('-c', 'import astropy; astropy.test()')


Test patch: