Repository: spack/spack

Issue:
bcl2fastq2
Hi! I'm having trouble installing bcl2fastq2 on macOS 10.14. All dependencies install normally, just the final install of the actual bcl2fastq2 package fails, already at the unpacking tarball. Any ideas what is going wrong?

```
$ uname -a 
Darwin Sara-MacBook.fritz.box 18.0.0 Darwin Kernel Version 18.0.0: Wed Aug 22 20:13:40 PDT 2018; root:xnu-4903.201.2~1/RELEASE_X86_64 x86_64
```

```
$./spack install bcl2fastq2

==> diffutils is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/diffutils-3.6-te2wy74ojq3v53ovt7s77jaxqxsmv3ly
==> bzip2 is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/bzip2-1.0.6-syohzw57v2jfag5du2x4bowziw3m5p67
==> zlib is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/zlib-1.2.11-uw6agnye4za7cpgyyaaejrzorhbhqno5
==> boost is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/boost-1.54.0-mfzhszy5w55xkrt56mvh3ft6ezqlqa6r
==> pkgconf is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/pkgconf-1.5.4-5lntwitdekpyj43kxfincaipd3fdgmtj
==> ncurses is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/ncurses-6.1-zvaa4lhlhilypw5quj3akyd3apbq5gap
==> readline is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/readline-7.0-722dzmgymxyxd6ovjvh4742kcetkqtfs
==> gdbm is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/gdbm-1.18.1-il2toejg5r3zjhw4tyodx7te5c3cb4is
==> perl is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/perl-5.26.2-uhov4vq4lk3u7biruczmxbytm22agi66
==> openssl is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/openssl-1.1.1-3imbiyu5y3deo44yyyx4bgdayg4ciemt
==> cmake is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/cmake-3.13.0-n5d7iyfajngmdo46ydpdnzwx43ui5nl7
==> libgpg-error is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libgpg-error-1.27-ef3dxjfx7myvx4xnbm5hwwoxqvv3kkz6
==> libgcrypt is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libgcrypt-1.8.1-zphnhsihwmmyktaoh52zjg5cyclj5zeu
==> xz is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/xz-5.2.4-i47twa3ynjmijdjvt4nag47wtg65m7jc
==> libxml2 is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libxml2-2.7.8-fnyhdqa4eoy7ueqihjlxacno43ttz67q
==> libiconv is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libiconv-1.15-u27q4rowrbactosi7n57ever5q44jddx
==> libxslt is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libxslt-1.1.26-hzudmqz4dxmkufdzusksmarvbo4tis7f
==> Installing bcl2fastq2
==> Using cached archive: /Users/sarawz/tools/spack/var/spack/cache/bcl2fastq2/bcl2fastq2-2.20.0.422.zip
==> Staging archive: /Users/sarawz/tools/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp/bcl2fastq2-v2-20-0-tar.zip
==> Created stage in /Users/sarawz/tools/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp
==> Unpacking bcl2fastq2 tarball
==> cwd sez: /private/tmp/root/spack-stage/spack-stage-kRycW1
==> Error: OSError: [Errno 66] Directory not empty: 'spack-expanded-archive'

/Users/sarawz/tools/spack/var/spack/repos/builtin/packages/bcl2fastq2/package.py:82, in wrap:
         79                    tar = which('tar')
         80                    tarball = os.path.basename(tarball)
         81                    tar('-xf', tarball)
  >>     82                    tty.msg("Finished unpacking bcl2fastq2 tarball")
```

```
$ ./spack spec --install-status bcl2fastq2
Input spec
--------------------------------
 -   bcl2fastq2

Concretized
--------------------------------
 -   bcl2fastq2@2.20.0.422%clang@10.0.0-apple patches=04956b22cdcc054c458a4ad4797fcec8a9c434755d0037517931031ba6a2e755,22bd1a5faa46013d0b0ceea4164b4cf688b180674c72b25b37ad7635897be081 arch=darwin-mojave-x86_64 
[+]      ^boost@1.54.0%clang@10.0.0-apple+atomic+chrono~clanglibcpp cxxstd=default +date_time~debug+exception+filesystem+graph~icu+iostreams+locale+log+math~mpi+multithreaded~numpy patches=f994ac84634f2f833a7a4d3179c5bf9a06f14349ef67aacba39d08837ffab004 ~pic+program_options~python+random+regex+serialization+shared+signals~singlethreaded+system~taggedlayout+test+thread+timer~versionedlayout+wave arch=darwin-mojave-x86_64 
[+]          ^bzip2@1.0.6%clang@10.0.0-apple+shared arch=darwin-mojave-x86_64 
[+]              ^diffutils@3.6%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]          ^zlib@1.2.11%clang@10.0.0-apple+optimize+pic+shared arch=darwin-mojave-x86_64 
[+]      ^cmake@3.13.0%clang@10.0.0-apple~doc+ncurses+openssl+ownlibs~qt arch=darwin-mojave-x86_64 
[+]          ^ncurses@6.1%clang@10.0.0-apple~symlinks~termlib arch=darwin-mojave-x86_64 
[+]              ^pkgconf@1.5.4%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]          ^openssl@1.1.1%clang@10.0.0-apple+systemcerts arch=darwin-mojave-x86_64 
[+]              ^perl@5.26.2%clang@10.0.0-apple+cpanm patches=0eac10ed90aeb0459ad8851f88081d439a4e41978e586ec743069e8b059370ac +shared+threads arch=darwin-mojave-x86_64 
[+]                  ^gdbm@1.18.1%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]                      ^readline@7.0%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]      ^libgcrypt@1.8.1%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]          ^libgpg-error@1.27%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]      ^libxml2@2.7.8%clang@10.0.0-apple~python arch=darwin-mojave-x86_64 
[+]          ^xz@5.2.4%clang@10.0.0-apple arch=darwin-mojave-x86_64 
[+]      ^libxslt@1.1.26%clang@10.0.0-apple~crypto arch=darwin-mojave-x86_64 
[+]          ^libiconv@1.15%clang@10.0.0-apple arch=darwin-mojave-x86_64 

```




Discussion:
Here's some extra output with `spack -d install bcl2fastq2` on macOS 18.2.0.

```
==> Installing bcl2fastq2
==> Searching for binary cache of bcl2fastq2
==> Warning: No Spack mirrors are currently configured
==> No binary for bcl2fastq2 found: installing from source
==> Deprecation warning: md5 checksums will not be supported in future Spack releases.
==> Using cached archive: /Users/eric/p/spack/var/spack/cache/bcl2fastq2/bcl2fastq2-2.20.0.422.zip
==> Deprecation warning: md5 checksums will not be supported in future Spack releases.
==> Staging archive: /Users/eric/p/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp/bcl2fastq2-v2-20-0-tar.zip
==> '/usr/bin/unzip' '-q' '/Users/eric/p/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp/bcl2fastq2-v2-20-0-tar.zip'
==> Created stage in /Users/eric/p/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp
==> Unpacking bcl2fastq2 tarball
==> cwd sez: /private/var/folders/x0/lnw3tqcj5qg_p9tk7m99kyh00000gn/T/eric/spack-stage/spack-stage-LzC9kY
==> Copying spack-expanded-archive/bcl2fastq2-v2.20.0.422-Source.tar.gz to .
==> Error: OSError: [Errno 66] Directory not empty: 'spack-expanded-archive'

/Users/eric/p/spack/var/spack/repos/builtin/packages/bcl2fastq2/package.py:82, in wrap:
         79                    tar = which('tar')
         80                    tarball = os.path.basename(tarball)
         81                    tar('-xf', tarball)
  >>     82                    tty.msg("Finished unpacking bcl2fastq2 tarball")


Traceback (most recent call last):
  File "/Users/eric/p/spack/lib/spack/spack/build_environment.py", line 725, in child_process
    return_value = function()
  File "/Users/eric/p/spack/lib/spack/spack/package.py", line 1437, in build_process
    self.do_patch()
  File "/Users/eric/p/spack/lib/spack/spack/package.py", line 976, in do_patch
    self.do_stage()
  File "/Users/eric/p/spack/var/spack/repos/builtin/packages/bcl2fastq2/package.py", line 65, in do_stage
  File "/Users/eric/p/spack/lib/spack/spack/package.py", line 931, in do_stage
    self.stage.expand_archive()
  File "/Users/eric/p/spack/var/spack/repos/builtin/packages/bcl2fastq2/package.py", line 79, in wrap
    tar = which('tar')
OSError: [Errno 66] Directory not empty: 'spack-expanded-archive'
```
Can you run this and see if it solves the problem?
```console
$ spack clean bcl2fastq2
$ spack install bcl2fastq2
```
I'm wondering if the stage got itself in a bad state.
No change in outcome in my environment.
@hartzell can you take a look at this?
The same result for me as well. 

```
Sara-MacBook:bin sarawz$ ./spack clean bcl2fastq2
==> Cleaning build stage [bcl2fastq2@2.20.0.422%clang@10.0.0-apple patches=04956b22cdcc054c458a4ad4797fcec8a9c434755d0037517931031ba6a2e755,22bd1a5faa46013d0b0ceea4164b4cf688b180674c72b25b37ad7635897be081 arch=darwin-mojave-x86_64 /mgl5ivv]
Sara-MacBook:bin sarawz$ ./spack install bcl2fastq2
==> diffutils is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/diffutils-3.6-te2wy74ojq3v53ovt7s77jaxqxsmv3ly
==> bzip2 is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/bzip2-1.0.6-syohzw57v2jfag5du2x4bowziw3m5p67
==> zlib is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/zlib-1.2.11-uw6agnye4za7cpgyyaaejrzorhbhqno5
==> boost is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/boost-1.54.0-mfzhszy5w55xkrt56mvh3ft6ezqlqa6r
==> pkgconf is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/pkgconf-1.5.4-5lntwitdekpyj43kxfincaipd3fdgmtj
==> ncurses is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/ncurses-6.1-zvaa4lhlhilypw5quj3akyd3apbq5gap
==> readline is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/readline-7.0-722dzmgymxyxd6ovjvh4742kcetkqtfs
==> gdbm is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/gdbm-1.18.1-il2toejg5r3zjhw4tyodx7te5c3cb4is
==> perl is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/perl-5.26.2-uhov4vq4lk3u7biruczmxbytm22agi66
==> openssl is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/openssl-1.1.1-3imbiyu5y3deo44yyyx4bgdayg4ciemt
==> cmake is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/cmake-3.13.0-n5d7iyfajngmdo46ydpdnzwx43ui5nl7
==> libgpg-error is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libgpg-error-1.27-ef3dxjfx7myvx4xnbm5hwwoxqvv3kkz6
==> libgcrypt is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libgcrypt-1.8.1-zphnhsihwmmyktaoh52zjg5cyclj5zeu
==> xz is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/xz-5.2.4-i47twa3ynjmijdjvt4nag47wtg65m7jc
==> libxml2 is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libxml2-2.7.8-fnyhdqa4eoy7ueqihjlxacno43ttz67q
==> libiconv is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libiconv-1.15-u27q4rowrbactosi7n57ever5q44jddx
==> libxslt is already installed in /Users/sarawz/tools/spack/opt/spack/darwin-mojave-x86_64/clang-10.0.0-apple/libxslt-1.1.26-hzudmqz4dxmkufdzusksmarvbo4tis7f
==> Installing bcl2fastq2
==> Searching for binary cache of bcl2fastq2
==> Warning: No Spack mirrors are currently configured
==> No binary for bcl2fastq2 found: installing from source
==> Using cached archive: /Users/sarawz/tools/spack/var/spack/cache/bcl2fastq2/bcl2fastq2-2.20.0.422.zip
==> Staging archive: /Users/sarawz/tools/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp/bcl2fastq2-v2-20-0-tar.zip
==> Created stage in /Users/sarawz/tools/spack/var/spack/stage/bcl2fastq2-2.20.0.422-mgl5ivvchxixqdbbbsw4xhztl3zdi4gp
==> Unpacking bcl2fastq2 tarball
==> cwd sez: /private/var/folders/ql/k05_yv953z5_6lb948_kd8cw0000gn/T/sarawz/spack-stage/spack-stage-9SxZeC
==> Error: OSError: [Errno 66] Directory not empty: 'spack-expanded-archive'

/Users/sarawz/tools/spack/var/spack/repos/builtin/packages/bcl2fastq2/package.py:82, in wrap:
         79                    tar = which('tar')
         80                    tarball = os.path.basename(tarball)
         81                    tar('-xf', tarball)
  >>     82                    tty.msg("Finished unpacking bcl2fastq2 tarball")
```
Yep (just back from holiday travels, will get to it ASAP).
I'm building.  FWIW, the problem is probably that there's something left in the unpacking directory that's causing [the `os.rmdir` at line 78](https://github.com/spack/spack/blob/develop/var/spack/repos/builtin/packages/bcl2fastq2/package.py#L78) to fail (it returns `OSError` when called on non-empty directories).  Will look to see what's going on (why now, on Macs?), the answer will probably be to switch to `shutil.rmtree` as described [in the `os.rmdir` man page](https://docs.python.org/2/library/os.html#os.rmdir).  Pythonic feedback welcome.
Still building, but hmmm, looks like [this commit](https://github.com/spack/spack/commit/73c978ddd99973e29f2ba42078b10455c1de5ca8#diff-d920b4e38599837ee806323c2b09b3ce) replaced a call to `shutil.move` with a call to `copy`, which leaves the original file in place and breaks the `os.rmdir`.

I'll confirm, then PR a fix.





Resolving patch:
diff --git a/var/spack/repos/builtin/packages/bcl2fastq2/package.py b/var/spack/repos/builtin/packages/bcl2fastq2/package.py
--- a/var/spack/repos/builtin/packages/bcl2fastq2/package.py
+++ b/var/spack/repos/builtin/packages/bcl2fastq2/package.py
@@ -5,6 +5,7 @@
 
 from spack import *
 import os
+import shutil
 import glob
 import llnl.util.tty as tty
 
@@ -71,11 +72,10 @@ def wrap():
                     tty.msg("The tarball has already been unpacked")
                 else:
                     tty.msg("Unpacking bcl2fastq2 tarball")
-                    tty.msg("cwd sez: {0}".format(os.getcwd()))
                     tarball = glob.glob(join_path('spack-expanded-archive',
                                         'bcl2fastq2*.tar.gz'))[0]
                     copy(tarball, '.')
-                    os.rmdir('spack-expanded-archive')
+                    shutil.rmtree('spack-expanded-archive')
                     tar = which('tar')
                     tarball = os.path.basename(tarball)
                     tar('-xf', tarball)


Test patch: