internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spack/spack

Issue:
Cannot create a mirror for git spec
First off: love this package manager, it is exactly what we had been looking for.

Secondly:
`spack mirror create git` produces the error:

```
==> Warning: Error while fetching git@2.8.1
  Cannot archive without matching extensions.
==> Successfully updated mirror in /mirror
  Archive stats:
    0    already present
    0    added
    1    failed to fetch.
==> Error: Failed downloads:
git@2.8.1
```

Which seems to be because it is expecting an extension on [this](https://github.com/LLNL/spack/blob/76d42b520069e15aa7cf0fa8cbfb3d83c9be7052/lib/spack/spack/fetch_strategy.py#L310) line.

But the tarball it downloads is: `$SPACK_ROOT/var/spack/stage/git-2.8.1-mdglrz6t3slmg7xkhdsq7ia3opouudxw/v2.8.1`

I would be happy to create a PR to fix this...



Discussion:
@mwilliammyers: Welcome to Spack, we're glad to have you!

Feel free to contribute a PR if you think you can fix it. If not, one of us should get around to it eventually. It seems like this bug should affect a lot of packages that use a Git URL.

So I got a chance to look into this more and I am not convinced that we need to be checking to see if the extensions match seeing as they will be check-summed... If we do, the best solution I could come up with would involve calling the [`downloaded_file_extension()`](https://github.com/LLNL/spack/blob/develop/lib/spack/spack/url.py#L143) method on the `url` class to get the expected extension. This seems like an unnecessary check to me though. I just didn't want to take out the check in case there is a good reason it is there seeing as this would affect a lot of URLs. As it stands right now I don't think any URL that uses GitHub's [archive scheme](https://developer.github.com/v3/repos/contents/#get-archive-link) will work. 

Hmm, the only reason I can think of that we would want to check the extension would be for #282. Not sure if that was ever fixed. Anyone else have any knowledge of `download_file_extension()`?


Resolving patch:
diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py
--- a/lib/spack/spack/fetch_strategy.py
+++ b/lib/spack/spack/fetch_strategy.py
@@ -307,9 +307,6 @@ def archive(self, destination):
         if not self.archive_file:
             raise NoArchiveFileError("Cannot call archive() before fetching.")
 
-        if not extension(destination) == extension(self.archive_file):
-            raise ValueError("Cannot archive without matching extensions.")
-
         shutil.move(self.archive_file, destination)
 
     @_needs_stage


Test patch:
Repository: spack/spack

Issue:
`py-h5py` should not depend on `hdf5~mpi`
`py-h5py` wraps the HDF5 library. It contains these lines:

``` Python
    depends_on('hdf5@1.8.4:+mpi', when='+mpi')
    depends_on('hdf5@1.8.4:~mpi', when='~mpi')
    depends_on('mpi', when='+mpi')
```

The first two of these should probably read instead

``` Python
    depends_on('hdf5@1.8.4:')
    depends_on('hdf5+mpi', when='+mpi')
```

This would allow using a plain `h5py` (which is `h5py~mpi`) with a `hdf5+mpi`. I don't think there is a reason to disallow this.



Discussion:
I'm ok with this change.  Want to submit a PR?

@adamjstewart?

Sounds fine to me!


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-h5py/package.py b/var/spack/repos/builtin/packages/py-h5py/package.py
--- a/var/spack/repos/builtin/packages/py-h5py/package.py
+++ b/var/spack/repos/builtin/packages/py-h5py/package.py
@@ -43,8 +43,8 @@ class PyH5py(Package):
     # Build dependencies
     depends_on('py-cython@0.19:')
     depends_on('pkg-config')
-    depends_on('hdf5@1.8.4:+mpi', when='+mpi')
-    depends_on('hdf5@1.8.4:~mpi', when='~mpi')
+    depends_on('hdf5@1.8.4:')
+    depends_on('hdf5+mpi', when='+mpi')
     depends_on('mpi', when='+mpi')
 
     # Build and runtime dependencies


Test patch:
Repository: spack/spack

Issue:
Option to "git submodule update  --init" at pull-time
Some packages will perform this additional git pull step during install. This creates issues when connectivity is limited. An option like this would mean the correct source is available in the mirror. 



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/fetch_strategy.py b/lib/spack/spack/fetch_strategy.py
--- a/lib/spack/spack/fetch_strategy.py
+++ b/lib/spack/spack/fetch_strategy.py
@@ -366,8 +366,8 @@ def fetch(self):
             try:
                 self.check()
             except ChecksumError:
-                # Future fetchers will assume they don't need to download if the
-                # file remains
+                # Future fetchers will assume they don't need to
+                # download if the file remains
                 os.remove(self.archive_file)
                 raise
 
@@ -517,6 +517,7 @@ def __init__(self, **kwargs):
         super(GitFetchStrategy, self).__init__(
             'git', 'tag', 'branch', 'commit', **forwarded_args)
         self._git = None
+        self.submodules = kwargs.get('submodules', False)
 
     @property
     def git_version(self):
@@ -595,6 +596,10 @@ def fetch(self):
                 self.git('pull', '--tags', ignore_errors=1)
                 self.git('checkout', self.tag)
 
+        # Init submodules if the user asked for them.
+        if self.submodules:
+            self.git('submodule', 'update', '--init')
+
     def archive(self, destination):
         super(GitFetchStrategy, self).archive(destination, exclude='.git')
 


Test patch:
Repository: spack/spack

Issue:
Cannot install py-numpy: "No module named setuptools"
```
==> Already downloaded /xfs1/eschnetter/spacktest/spack/var/spack/stage/py-numpy-1.11.0-urymgx5gdatropsqd2ehdfjoln44isa6/py-numpy-1.11.0.tar.gz
==> Already staged py-numpy-1.11.0-urymgx5gdatropsqd2ehdfjoln44isa6 in /xfs1/eschnetter/spacktest/spack/var/spack/stage/py-numpy-1.11.0-urymgx5gdatropsqd2ehdfjoln44isa6
==> No patches needed for py-numpy
==> Building py-numpy
==> '/xfs1/eschnetter/spacktest/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.1.0-spacktest/python-2.7.12-b5sf3vacbqfghadqm6uj2tti2oemd3xy/bin/python' 'setup.py' 'install' '--prefix=/xfs1/eschnetter/spacktest/spack/opt/spack/linux-Ubuntu14-x86_64/gcc-6.1.0-spacktest/py-numpy-1.11.0-urymgx5gdatropsqd2ehdfjoln44isa6'
Running from numpy source directory.

Note: if you need reliable uninstall behavior, then install
with pip instead of using `setup.py install`:

  - `pip install .`       (from a git repo or downloaded source
                           release)
  - `pip install numpy`   (last Numpy release on PyPi)


Traceback (most recent call last):
  File "setup.py", line 386, in <module>
    setup_package()
  File "setup.py", line 363, in setup_package
    from setuptools import setup
ImportError: No module named setuptools
```

I used this package definition for an umbrella package:

```
from spack import *

class Umbrella(Package):
    homepage = "Umbrella package"
    # This is just a dummy download
    url = "http://zlib.net/zlib-1.2.8.tar.gz"
    version('1.2.8', '44d667c142d7cda120332623eab69f40')

    depends_on("boost +mpi")
    depends_on("fftw +mpi +openmp")
    depends_on("git")
    depends_on("gsl")
    depends_on("hdf5 +mpi")
    depends_on("hdf5-blosc")
    depends_on("hwloc")
    depends_on("jemalloc")
    depends_on("julia")
    # depends_on("julia +hdf5 +mpi")
    depends_on("llvm")
    depends_on("lmod")
    depends_on("openblas +openmp")
    depends_on("openmpi")
    depends_on("papi")
    depends_on("petsc +boost +hdf5 +mpi")
    depends_on("py-h5py +mpi")
    depends_on("py-mpi4py")
    depends_on("py-numpy")
    depends_on("py-scipy")
    # Specify version @2.7.12 explicitly to circumvent a concretization bug
    depends_on("python @2.7.12")
    depends_on("qthreads")
    depends_on("swig")
    depends_on("tmux")

    def install(self, spec, prefix):
        # This package does not install anything; it only installs its
        # dependencies
        mkdirp(prefix.lib)
```



Discussion:
Try adding `depends_on('py-setuptools')` to py-numpy and see if that solves the problem for you.

Yes, this corrects the error.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/py-numpy/package.py b/var/spack/repos/builtin/packages/py-numpy/package.py
--- a/var/spack/repos/builtin/packages/py-numpy/package.py
+++ b/var/spack/repos/builtin/packages/py-numpy/package.py
@@ -44,6 +44,7 @@ class PyNumpy(Package):
 
     extends('python')
     depends_on('py-nose', type='build')
+    depends_on('py-setuptools')
     depends_on('blas',   when='+blas')
     depends_on('lapack', when='+lapack')
 


Test patch:
Repository: spack/spack

Issue:
R module (and dotkit?) file needs LIBRARY_PATH and LD_LIBRARY_PATH for bioconductor installs
Currently the module file setup by spack for R will not add the correct library paths which results in broken library links.

$ ldd ~/R/x86_64-pc-linux-gnu-library/3.3/ncdf4/libs/ncdf4.so  | grep -i not
        libR.so => not found

If the library paths are added to the environment then ncdf4.so dynamically links correctly.
$  ldd R/x86_64-pc-linux-gnu-library/3.3/ncdf4/libs/ncdf4.so | grep libR.so
        libR.so => /gpfs/software/spack/opt/spack/linux-redhat6-x86_64/gcc-6.1.0/R-3.3.1-tpkh25pakjkfxwqth7kp3gxl4grlf6c2/rlib/R/lib/libR.so (0x00002aaaae244000)

Need to add the following to the R module file:
prepend-path --delim ":" LIBRARY_PATH "PATH_TO_R_library"
prepend-path --delim ":" LD_LIBRARY_PATH "PATH_TO_R_library"

ncdf4 was installed as a pre-req to GENESIS. https://www.bioconductor.org/packages/devel/bioc/html/GENESIS.html

I will look into this further to submit a patch as I get my user running.



Discussion:
it could have to do with the absence of `soname`/`install_name` in `libR`, which should be fixed. My understanding is that otherwise `netcdf` will be dynamically linking against anything it can find in paths, not necessarily something installed by Spack.

@adamhough You can customize what gets written in module files, see [the documentation here](http://software.llnl.gov/spack/basic_usage.html#configuration-files).

Possibly that would also explain why I am picking up modules from other packages not in my listed library paths which is because of netcdf.  That would also explain why ncdf4.so is also picking up libscif from torque even though it is not in any default path in my build environment.

@alalazo Thanks for pointing that out as I had forgotten about that.  Only started using spack last week.

@adamhough Not a problem!


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/R/package.py b/var/spack/repos/builtin/packages/R/package.py
--- a/var/spack/repos/builtin/packages/R/package.py
+++ b/var/spack/repos/builtin/packages/R/package.py
@@ -136,6 +136,14 @@ def setup_dependent_environment(self, spack_env, run_env, extension_spec):
             run_env.prepend_path('R_LIBS', os.path.join(
                 extension_spec.prefix, self.r_lib_dir))
 
+    def setup_environment(self, spack_env, run_env):
+        run_env.prepend_path('LIBRARY_PATH',
+                             join_path(self.prefix, 'rlib', 'R', 'lib'))
+        run_env.prepend_path('LD_LIBRARY_PATH',
+                             join_path(self.prefix, 'rlib', 'R', 'lib'))
+        run_env.prepend_path('CPATH',
+                             join_path(self.prefix, 'rlib', 'R', 'include'))
+
     def setup_dependent_package(self, module, ext_spec):
         """Called before R modules' install() methods. In most cases,
         extensions will only need to have one line:


Test patch:
Repository: spack/spack

Issue:
openssl package not fetching
The changes in #1333 have caused the fetch url to be incorrect.

```
spack fetch openssl
==> Trying to fetch from file:///home/gjohnson/spack/var/spack/cache/openssl/openssl-1.0.2h.tar.gz
curl: (37) Couldn't open file /home/gjohnson/spack/var/spack/cache/openssl/openssl-1.0.2h.tar.gz
==> Fetching from file:///home/gjohnson/spack/var/spack/cache/openssl/openssl-1.0.2h.tar.gz failed.
==> Trying to fetch from https://www.openssl.org/source/openssl-openssl@1.0.2h%gcc@4.8.5 arch=linux-centos7-x86_64^zlib@1.2.8%gcc@4.8.5 arch=linux-centos7-x86_64.tar.gz

curl: (22) The requested URL returned error: 400 Bad Request
==> Fetching from https://www.openssl.org/source/openssl-openssl@1.0.2h%gcc@4.8.5 arch=linux-centos7-x86_64^zlib@1.2.8%gcc@4.8.5 arch=linux-centos7-x86_64.tar.gz failed.
==> Error: All fetchers failed for openssl-1.0.2h-xoai5allk33en37vtec3untfc6h6xifs
```

It is putting the entire spec in the url.



Discussion:
Gimme a minute, I'll submit a PR to fix it.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/openssl/package.py b/var/spack/repos/builtin/packages/openssl/package.py
--- a/var/spack/repos/builtin/packages/openssl/package.py
+++ b/var/spack/repos/builtin/packages/openssl/package.py
@@ -52,7 +52,7 @@ def url_for_version(self, version):
         if '@system' in self.spec:
             return '@system (reserved version for system openssl)'
         else:
-            return super(Openssl, self).url_for_version(self.spec)
+            return super(Openssl, self).url_for_version(self.version)
 
     def handle_fetch_error(self, error):
         tty.warn("Fetching OpenSSL failed. This may indicate that OpenSSL has "


Test patch:
Repository: spack/spack

Issue:
JDK install problem: AttributeError: 'module' object has no attribute 'dir_util'
I'm cleaning up my nextflow package (which I haven't touched in days).  Now I can't install the jdk.

I've rebased onto the tip of develop, and I'm using the python that spack built instead of CentOS 7's.  Not sure what else changed.

Here's what I see:

```
[throgg@spack-central spack]$ spack install nextflow
==> Installing nextflow
==> Installing jdk
==> [Jdk] Adding required commandline options to curl before performing fetch: ['-j', '-H', 'Cookie: oraclelicense=accept-securebackup-cookie']
==> Already downloaded /home/throgg/spack/var/spack/stage/jdk-8u92-linux-x64-fzug57sw2sdor4dsjooesubqttyk43a7/jdk-8u92-linux-x64.tar.gz
==> Already staged jdk-8u92-linux-x64-fzug57sw2sdor4dsjooesubqttyk43a7 in /home/throgg/spack/var/spack/stage/jdk-8u92-linux-x64-fzug57sw2sdor4dsjooesubqttyk43a7
==> No patches needed for jdk
==> Building jdk
Traceback (most recent call last):
  File "/home/throgg/spack/lib/spack/spack/build_environment.py", line 512, in fork
    function()
  File "/home/throgg/spack/lib/spack/spack/package.py", line 997, in build_process
    self.install(self.spec, self.prefix)
  File "/home/throgg/spack/var/spack/repos/builtin/packages/jdk/package.py", line 70, in install
    distutils.dir_util.copy_tree(".", prefix)
AttributeError: 'module' object has no attribute 'dir_util'
==> Error: Installation process had nonzero exit code : 256
```

This change fixes it, but usual Python newbie disclaimers apply:

```
--- a/var/spack/repos/builtin/packages/jdk/package.py
+++ b/var/spack/repos/builtin/packages/jdk/package.py
@@ -26,6 +26,7 @@
 # Author: Justin Too <too1@llnl.gov>
 #
 import distutils
+from distutils.dir_util import copy_tree

 import spack
 from spack import *
@@ -67,4 +68,4 @@ class Jdk(Package):
         super(Jdk, self).do_fetch(mirror_only)

     def install(self, spec, prefix):
-        distutils.dir_util.copy_tree(".", prefix)
+        copy_tree(".", prefix)
```

Is this a Real Bug(tm) or something I did?  If its a bug, I can PR it.



Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/jdk/package.py b/var/spack/repos/builtin/packages/jdk/package.py
--- a/var/spack/repos/builtin/packages/jdk/package.py
+++ b/var/spack/repos/builtin/packages/jdk/package.py
@@ -25,7 +25,7 @@
 #
 # Author: Justin Too <too1@llnl.gov>
 #
-import distutils
+import distutils.dir_util
 
 import spack
 from spack import *


Test patch:
Repository: spack/spack

Issue:
Cannot fetch git repo
#1346 looks like it may have changed URL fetching. @glennpj can you take a look at this? Examples from http://software.llnl.gov/spack/packaging_guide.html#git-fetch won't work.



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/stage.py b/lib/spack/spack/stage.py
--- a/lib/spack/spack/stage.py
+++ b/lib/spack/spack/stage.py
@@ -318,10 +318,11 @@ def fetch(self, mirror_only=False):
             fetchers.insert(0, spack.cache.fetcher(self.mirror_path, digest))
 
             # Look for the archive in list_url
-            archive_version = spack.url.parse_version(self.default_fetcher.url)
             package_name = os.path.dirname(self.mirror_path)
             pkg = spack.repo.get(package_name)
             if pkg.list_url is not None and pkg.url is not None:
+                archive_version = spack.url.parse_version(
+                    self.default_fetcher.url)
                 versions = pkg.fetch_remote_versions()
                 try:
                     url_from_list = versions[Version(archive_version)]


Test patch:
Repository: spack/spack

Issue:
Typo in UnavailableCompilerVersionError()
Typo causes a TypeError to be thown whenever a specified compiler doesn't exist for the arch.

```
...
 File "/autofs/nccs-svm1_home2/wyn/spack-titan/lib/spack/spack/concretize.py", line 498, in __init__
    "No available compiler version matches '%s' on operating_system %s" % compiler_spec, operating_system,  # NOQA: ignore=E501
TypeError: not enough arguments for format string
```

Typo in UnavailableCompilerVersionError()
Typo causes a TypeError to be thown whenever a specified compiler doesn't exist for the arch.

```
...
 File "/autofs/nccs-svm1_home2/wyn/spack-titan/lib/spack/spack/concretize.py", line 498, in __init__
    "No available compiler version matches '%s' on operating_system %s" % compiler_spec, operating_system,  # NOQA: ignore=E501
TypeError: not enough arguments for format string
```



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/concretize.py b/lib/spack/spack/concretize.py
--- a/lib/spack/spack/concretize.py
+++ b/lib/spack/spack/concretize.py
@@ -495,7 +495,7 @@ class UnavailableCompilerVersionError(spack.error.SpackError):
 
     def __init__(self, compiler_spec, operating_system):
         super(UnavailableCompilerVersionError, self).__init__(
-            "No available compiler version matches '%s' on operating_system %s" % compiler_spec, operating_system,  # NOQA: ignore=E501
+            "No available compiler version matches '%s' on operating_system %s" % (compiler_spec, operating_system),  # NOQA: ignore=E501
             "Run 'spack compilers' to see available compiler Options.")
 
 


Test patch:
Repository: spack/spack

Issue:
DeprecationWarning in repository.py
```
/blues/gpfs/home/software/spack-0.9.1/lib/spack/spack/repository.py:81: DeprecationWarning: object.__init__() takes no parameters
  super(ModuleType, self).__init__(self, namespace)
```

It seems like the super class of ModuleType is object, so there must be a bug in here. @tgamblin, it seems like you're the only one to edit this file recently.



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/repository.py b/lib/spack/spack/repository.py
--- a/lib/spack/spack/repository.py
+++ b/lib/spack/spack/repository.py
@@ -78,7 +78,7 @@ def converter(self, spec_like, *args, **kwargs):
 class SpackNamespace(ModuleType):
     """ Allow lazy loading of modules."""
     def __init__(self, namespace):
-        super(ModuleType, self).__init__(self, namespace)
+        super(SpackNamespace, self).__init__(namespace)
         self.__file__ = "(spack namespace)"
         self.__path__ = []
         self.__name__ = namespace


Test patch:
Repository: spack/spack

Issue:
fenics package is broken
For some reason the download urls generated by the package seems to contain newlines and/or other whitespace. It also seems like a part of the url is duplicated.

``` console
$ spack checksum fenics
==> Found 5 versions of fenics
  2016.1.0  https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz
  1.6.0     https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-1.6.0.tar.gz
  1.5.0     https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-1.5.0.tar.gz
  1.4.0     https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-1.4.0.tar.gz
  1.3.0     https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-1.3.0.tar.gz

How many would you like to checksum? (default is 5, q to abort) 1
==> Downloading...
==> Trying to fetch from https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz
curl: (3) Illegal characters found in URL
==> Fetching from https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz failed.
==> Something failed on https://bitbucket.org/fenics-project/dolfin/
                    /fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz, skipping.
    (All fetchers failed for spack-stage-5eh1n5)
==> Error: Could not fetch any versions for fenics
```



Discussion:
I took a look at this but I don't see anything fishy in the package.py. At first I thought it was because the URL was too long, but I couldn't reproduce it in other packages. Then I thought it was because the package had resources, but other packages with resources don't have this problem. Then I thought we were overriding some `base_url` variable in Package, but there isn't a `base_url` variable in any Spack library file. I'm stumped. Did this ever work before? @nrichart?

I never checked the `spack checksum` on this package so I cannot tell if it worked. The strange think is that it downloads correctly.

After further investigation it comes from the web page https://bitbucket.org/fenics-project/dolfin/downloads

```
<td class="name"><a class="execute" href="
    /fenics-project/dolfin/downloads/dolfin-2016.1.0.tar.gz.asc">dolfin-2016.1.0.tar.gz.asc</a>
</td>
```

The `\n` is not stripped by spack


Resolving patch:
diff --git a/lib/spack/spack/util/web.py b/lib/spack/spack/util/web.py
--- a/lib/spack/spack/util/web.py
+++ b/lib/spack/spack/util/web.py
@@ -109,7 +109,7 @@ def _spider(args):
 
         while link_parser.links:
             raw_link = link_parser.links.pop()
-            abs_link = urlparse.urljoin(response_url, raw_link)
+            abs_link = urlparse.urljoin(response_url, raw_link.strip())
 
             links.add(abs_link)
 


Test patch: