internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: spack/spack

Issue:
Document Spack Resources
@tgamblin @alalazo @adamjstewart

I cannot find documentation on Spack resources mechanism at this point.  We need this documentation before it can be considered a feature we wish people to use.



Discussion:
@davydden 

@becker33 @tgamblin If you assign this to me (just as a reminder), I'll try to write something up asap. Any preference where to put the docs ?

Thanks! I would suggest putting it as a new section in the Packaging Guide.

On Wed, Aug 17, 2016 at 9:34 AM, Massimiliano Culpo <
notifications@github.com> wrote:

> @becker33 https://github.com/becker33 @tgamblin
> https://github.com/tgamblin If you assign this to me (just as a
> reminder), I'll try to write something up asap. Any preference where to put
> the docs ?
> 
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> https://github.com/LLNL/spack/issues/1551#issuecomment-240412118, or mute
> the thread
> https://github.com/notifications/unsubscribe-auth/AB1cd_D2shrQ50lRpyqvL7KqfJPnuvDqks5qgw3QgaJpZM4JmcsF
> .

This should probably go in the packaging guide after VCS fetching

Thanks!


Resolving patch:
diff --git a/lib/spack/spack/directives.py b/lib/spack/spack/directives.py
--- a/lib/spack/spack/directives.py
+++ b/lib/spack/spack/directives.py
@@ -292,17 +292,17 @@ def resource(pkg, **kwargs):
     Define an external resource to be fetched and staged when building the
     package. Based on the keywords present in the dictionary the appropriate
     FetchStrategy will be used for the resource. Resources are fetched and
-    staged in their own folder inside spack stage area, and then linked into
+    staged in their own folder inside spack stage area, and then moved into
     the stage area of the package that needs them.
 
     List of recognized keywords:
 
     * 'when' : (optional) represents the condition upon which the resource is
     needed
-    * 'destination' : (optional) path where to link the resource. This path
+    * 'destination' : (optional) path where to move the resource. This path
     must be relative to the main package stage area.
     * 'placement' : (optional) gives the possibility to fine tune how the
-    resource is linked into the main package stage area.
+    resource is moved into the main package stage area.
     """
     when = kwargs.get('when', pkg.name)
     destination = kwargs.get('destination', "")


Test patch:
Repository: spack/spack

Issue:
New Package: git-lfs
Has anyone ever looked at adding a [git-lfs](https://git-lfs.github.com) package?

Specifically, for the client side support. This is a missing piece for LLNL LC users that might be thinking about using BitBucket's Git LFS support.



Discussion:
judging from Homebrew's formula https://github.com/Homebrew/homebrew-core/blob/master/Formula/git-lfs.rb , it should be extremely straight-forward to compile:

```
depends_on "go" => :build

  def install
    system "./script/bootstrap"
    bin.install "bin/git-lfs"
  end
```


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/git-lfs/package.py b/var/spack/repos/builtin/packages/git-lfs/package.py
new file mode 100644
--- /dev/null
+++ b/var/spack/repos/builtin/packages/git-lfs/package.py
@@ -0,0 +1,42 @@
+##############################################################################
+# Copyright (c) 2013-2016, Lawrence Livermore National Security, LLC.
+# Produced at the Lawrence Livermore National Laboratory.
+#
+# This file is part of Spack.
+# Created by Todd Gamblin, tgamblin@llnl.gov, All rights reserved.
+# LLNL-CODE-647188
+#
+# For details, see https://github.com/llnl/spack
+# Please also see the LICENSE file for our notice and the LGPL.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU Lesser General Public License (as
+# published by the Free Software Foundation) version 2.1, February 1999.
+#
+# This program is distributed in the hope that it will be useful, but
+# WITHOUT ANY WARRANTY; without even the IMPLIED WARRANTY OF
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the terms and
+# conditions of the GNU Lesser General Public License for more details.
+#
+# You should have received a copy of the GNU Lesser General Public
+# License along with this program; if not, write to the Free Software
+# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
+##############################################################################
+from spack import *
+
+
+class GitLfs(Package):
+    """Tool for managing large files with Git."""
+
+    homepage = "https://git-lfs.github.com"
+    url      = "https://github.com/github/git-lfs/archive/v1.4.1.tar.gz"
+
+    version('1.4.1', 'c62a314d96d3a30af4d98fa3305ad317')
+
+    depends_on('go@1.5:', type='build')
+    depends_on('git@1.8.2:', type='run')
+
+    def install(self, spec, prefix):
+        bootstrap = Executable('./scripts/bootstrap')
+        bootstrap()
+        install('bin/git-lfs', prefix.bin)


Test patch:
Repository: spack/spack

Issue:
[Bug] Missing constructor arguments to class SpackYAMLError
@tgamblin 

See below.  `SpackYAMLError` requires two constructor args, but the last two instantiations of it below have just one.  That causes another exception while trying to throw the first.

```
    @staticmethod
    def from_yaml(stream):
        try:
            yfile = yaml.load(stream)
        except MarkedYAMLError, e:
            raise spack.spec.SpackYAMLError(
                "error parsing YAML ProviderIndex cache:", str(e))

        if not isinstance(yfile, dict):
            raise spack.spec.SpackYAMLError(
                "YAML ProviderIndex was not a dict.")

        if 'provider_index' not in yfile:
            raise spack.spec.SpackYAMLError(
                "YAML ProviderIndex does not start with 'provider_index'")
```



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/provider_index.py b/lib/spack/spack/provider_index.py
--- a/lib/spack/spack/provider_index.py
+++ b/lib/spack/spack/provider_index.py
@@ -32,6 +32,7 @@
 from yaml.error import MarkedYAMLError
 
 import spack
+import spack.error
 
 
 class ProviderIndex(object):
@@ -201,11 +202,10 @@ def from_yaml(stream):
                 "error parsing YAML ProviderIndex cache:", str(e))
 
         if not isinstance(yfile, dict):
-            raise spack.spec.SpackYAMLError(
-                "YAML ProviderIndex was not a dict.")
+            raise ProviderIndexError("YAML ProviderIndex was not a dict.")
 
         if 'provider_index' not in yfile:
-            raise spack.spec.SpackYAMLError(
+            raise ProviderIndexError(
                 "YAML ProviderIndex does not start with 'provider_index'")
 
         index = ProviderIndex()
@@ -291,3 +291,7 @@ def mapiter(mappings):
         (name, out_mapping_type([
             transform_fun(vpkg, pset) for vpkg, pset in mapiter(mappings)]))
         for name, mappings in providers.items())
+
+
+class ProviderIndexError(spack.error.SpackError):
+    """Raised when there is a problem with a ProviderIndex."""


Test patch:
Repository: spack/spack

Issue:
Cannot download lzo
Since today or yesterday I see this error message on a clean Ubuntu 16.04 Spack install:

```
curl: (37) Couldn't open file /home/spacktest/spacktest/spack/var/spack/cache/lzo/lzo-2.09.tar.gz
curl: (60) server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none
More details here: http://curl.haxx.se/docs/sslcerts.html
```

Is this a problem with the lzo package that points to a dysfunctional server? Or do we just need to wait for a few days for Ubuntu to update its certificates?



Discussion:

Resolving patch:
diff --git a/var/spack/repos/builtin/packages/lzo/package.py b/var/spack/repos/builtin/packages/lzo/package.py
--- a/var/spack/repos/builtin/packages/lzo/package.py
+++ b/var/spack/repos/builtin/packages/lzo/package.py
@@ -29,7 +29,7 @@ class Lzo(Package):
     """Real-time data compression library"""
 
     homepage = 'https://www.oberhumer.com/opensource/lzo/'
-    url = 'https://www.oberhumer.com/opensource/lzo/download/lzo-2.09.tar.gz'
+    url = 'http://www.oberhumer.com/opensource/lzo/download/lzo-2.09.tar.gz'
 
     version('2.09', 'c7ffc9a103afe2d1bba0b015e7aa887f')
     version('2.08', 'fcec64c26a0f4f4901468f360029678f')


Test patch:
Repository: spack/spack

Issue:
TypeError when trying to install hoomd-blue
I'm getting this stack trace when I try and run `bin/spack -vd install hoomd-blue@2.0.3~doc%gcc@4.9.3 arch=cray-CNL-istanbul`

```
  File "/autofs/nccs-svm1_home2/wyn/spack/lib/spack/spack/provider_index.py", line 235, in merge
    spdict[provided_spec] += opdict[provided_spec]
TypeError: unsupported operand type(s) for +=: 'set' and 'set'
```

I'm not 100% sure on what's triggering that since it worked last week, but when I cloned a fresh copy of spack, it caused this issue. Since it's an operation on two sets, I tried to change the line to `spdict[provided_spec] = spdict[provided_spec].union(opdict[provided_spec])` And it seems to not fail out at that point anymore. Could someone more familiar with spack's internals make sure that was the correct fix?



Discussion:

Resolving patch:
diff --git a/lib/spack/spack/provider_index.py b/lib/spack/spack/provider_index.py
--- a/lib/spack/spack/provider_index.py
+++ b/lib/spack/spack/provider_index.py
@@ -232,7 +232,8 @@ def merge(self, other):
                     spdict[provided_spec] = opdict[provided_spec]
                     continue
 
-                spdict[provided_spec] += opdict[provided_spec]
+                spdict[provided_spec] = \
+                    spdict[provided_spec].union(opdict[provided_spec])
 
     def remove_provider(self, pkg_name):
         """Remove a provider from the ProviderIndex."""


Test patch:
Repository: spack/spack

Issue:
Feature request: Can we have an 'extra verbose' option for install?
I frequently need to debug a failed build in spack.  At least for cmake-based packages, this would be much easier if the project was configured with `-DCMAKE_VERBOSE_MAKEFILE=ON` or equivalently, make was run with the option `VERBOSE=ON`.  Could one of these be the default?  This would save the detailed build instructions into spack-build.out, which is useful IMHO.  If that is not reasonable, maybe we could have a `-V` (very verbose) option for `spack install.`

What do you think?

Are there similar tricks that can be played with other build tools to increase their output verbosity?



Discussion:
I vote for `-DCMAKE_VERBOSE_MAKEFILE:BOOL=ON` to be added to `std_cmake_args`

@KineticTheory: we should have that stuff in the `std_cmake_args` in `build_environment.py` as @alalazo says.  I think it's an oversight that it's not there already.  Then it'll be on for all builds and it will get put in the `build.out` file, too.


Resolving patch:
diff --git a/lib/spack/spack/build_environment.py b/lib/spack/spack/build_environment.py
--- a/lib/spack/spack/build_environment.py
+++ b/lib/spack/spack/build_environment.py
@@ -408,7 +408,8 @@ def get_rpaths(pkg):
 def get_std_cmake_args(cmake_pkg):
     # standard CMake arguments
     ret = ['-DCMAKE_INSTALL_PREFIX=%s' % cmake_pkg.prefix,
-           '-DCMAKE_BUILD_TYPE=RelWithDebInfo']
+           '-DCMAKE_BUILD_TYPE=RelWithDebInfo',
+           '-DCMAKE_VERBOSE_MAKEFILE=ON']
     if platform.mac_ver()[0]:
         ret.append('-DCMAKE_FIND_FRAMEWORK=LAST')
 


Test patch:
Repository: spack/spack

Issue:
`spack checksum` finds wrong URLs
I was having some problems getting spack to find the correct URL for files.

With these settings

``` python
    homepage = "http://fishshell.com/"
    url      = "http://fishshell.com/files/2.2.0/fish-2.2.0.tar.gz"
    list_url = homepage
```

I get the following result (with wrong URLs):

``` sh
$ spack checksum fish
==> Found 5 versions of fish.
  2.2.0     http://fishshell.com/fish-2.2.0.tar.gz
  2.1.2     http://fishshell.com/fish-2.1.2.tar.gz
  2.1.1     http://fishshell.com/fish-2.1.1.tar.gz
  2.1.0     http://fishshell.com/fish-2.1.0.tar.gz
  2.0.0     http://fishshell.com/fish-2.0.0.tar.gz

How many would you like to checksum? (default is 5, q to abort)
==> Downloading...
==> Trying to fetch from http://fishshell.com/fish-2.2.0.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
==> Fetching from http://fishshell.com/fish-2.2.0.tar.gz failed.
==> Error: All fetchers failed for spack-stage-Slflbn
```



Discussion:
I have (partially) resolved this issue with pull request #408. The reason why Spack couldn't download http://fishshell.com/fish-2.2.0.tar.gz is because that URL does not exist. By setting list_url and list_depth properly, Spack is now able to download any version of Fish that you want.

``` bash
$ spack fetch fish@2.1.0
==> Warning: There is no checksum on file to fetch fish@2.1.0 safely.
  Fetch anyway? [y/N] y
==> Fetching with no checksum.
  Add a checksum or use --no-checksum to skip this check.
==> Trying to fetch from http://fishshell.com/files/2.1.0/fish-2.1.0.tar.gz
######################################################################## 100.0%
```

However, this does not solve the issue that Spack shouldn't find any remote versions at that location. With the fix in place, `spack versions` and `spack checksum` do not find any remote versions, although they should. But `spack install` and `spack fetch` now work, which I think is a lot more important.

Does anyone have any idea why Spack was previously finding other versions of Fish but at the wrong URL? Now it doesn't find any other versions of Fish but at the right URL. I have noticed that Spack has a lot of trouble finding versions of packages as soon as list_url and list_depth are necessary.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/fish/package.py b/var/spack/repos/builtin/packages/fish/package.py
--- a/var/spack/repos/builtin/packages/fish/package.py
+++ b/var/spack/repos/builtin/packages/fish/package.py
@@ -32,8 +32,7 @@ class Fish(Package):
 
     homepage = "http://fishshell.com/"
     url      = "http://fishshell.com/files/2.2.0/fish-2.2.0.tar.gz"
-    list_url = "http://fishshell.com/files/"
-    list_depth = 2
+    list_url = "http://fishshell.com/"
 
     version('2.2.0', 'a76339fd14ce2ec229283c53e805faac48c3e99d9e3ede9d82c0554acfc7b77a')
 


Test patch:
Repository: spack/spack

Issue:
Error: Class constructor failed for package 'cdd'.
I'm exploring `spack mirror create`.  I just naively tried to run it to see what it would do with the things I have installed, which does **not** include _cdd_, and ran into this:

```
[hartzelg@rss-spack spack]$ spack -v mirror create
==> Error: Class constructor failed for package 'cdd'.

Caused by:
TypeError: 'str' object is not callable
  File "/rss/spack/lib/spack/spack/repository.py", line 584, in get
    self._instances[key] = package_class(copy)
  File "/rss/spack/lib/spack/spack/package.py", line 398, in __init__
    f = fs.for_package_version(self, self.version)
  File "/rss/spack/lib/spack/spack/fetch_strategy.py", line 852, in for_package_version
    attrs['url'] = pkg.url_for_version(version)
  File "/rss/spack/var/spack/repos/builtin/packages/cdd/package.py", line 40, in url_for_version
    str(version.dotted()).replace('.', ''))

[hartzelg@rss-spack spack]$ spack find
```

Before I dig into line 40 of `.../cdd/package.py`, does this ring any bells for anyone?



Discussion:
`spack mirror create` with no args actually mirrors _everything_, not what is currently installed, so be careful.  That's probably why you're getting this error in `cdd`.

Currently installed stuff seems like a better default, to be honest.  That would be an easy fix in `cmd/mirror.py`

> That's probably why you're getting this error in cdd

Well, that explains why I'm seeing a problem with _cdd_ even though I don't have it installed.

It does _not_ explain why _cdd_ is exploding (though I guess I don't care...), does it?

I believe `dotted` is a property, not a function call, so removing the `()` should fix this.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/cdd/package.py b/var/spack/repos/builtin/packages/cdd/package.py
--- a/var/spack/repos/builtin/packages/cdd/package.py
+++ b/var/spack/repos/builtin/packages/cdd/package.py
@@ -37,7 +37,7 @@ class Cdd(Package):
 
     def url_for_version(self, version):
         return ("ftp://ftp.ifor.math.ethz.ch/pub/fukuda/cdd/cdd-%s.tar.gz" %
-                str(version.dotted()).replace('.', ''))
+                str(version.dotted).replace('.', ''))
 
     version('0.61a', '22c24a7a9349dd7ec0e24531925a02d9')
 


Test patch:
Repository: spack/spack

Issue:
spack patches for gcc installation (version 4.9.x)
I was trying to install `gcc` and see following : 

```bash
○ → spack purge -a
○ → spack install -v gcc@4.9.4 %gcc@4.4.7        #same for 4.9.3
....
==> Trying to fetch from http://ftp.gnu.org/gnu/gcc/gcc-4.9.4/gcc-4.9.4.tar.bz2
######################################################################## 100.0%
==> Staging archive: /some-gpfs-path/spack/var/spack/stage/gcc-4.9.4-ybvygharx7lyngmbsgb224cqm65jinib/gcc-4.9.4.tar.bz2
==> Created stage in /some-gpfs-path/spack/var/spack/stage/gcc-4.9.4-ybvygharx7lyngmbsgb224cqm65jinib
Reversed (or previously applied) patch detected!  Assume -R? [n]
```

I haven't looked at which `patches` but wondering if those are backported or something else.


Discussion:
Must be. Can you see which patch it was? We just need to modify the version range to exclude 4.9.4.

And for `5.2.0`

``` bash

$ spack install -v gcc@5.2.0 %gcc@4.4.7

libtool: compile:  /tmp/kumbhar/spack-stage/spack-stage-MgPXZ9/spack-build/./gcc/xgcc -B/tmp/kumbhar/spack-stage/spack-stage-MgPXZ9/spack-build/./gcc/ -B/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/opt/spack/linux-redhatenterpriseserver6-x86_64/gcc-4.4.7/gcc-5.2.0-3lxzv5lb6wttupnukbfrztgipznktc3a/x86_64-unknown-linux
....
/gpfs/bbp.cscs.ch/home/kumbhar/workarena/systems/lugviz/softwares/sources/spack/var/spack/stage/gcc-5.2.0-3lxzv5lb6wttupnukbfrztgipznktc3a/gcc-5.2.0/libgfortran/generated/cshift1_4.c  -fPIC -DPIC -o .libs/cshift1_4.o
./libtool: line 1158: /tmp/kumbhar/spack-stage/spack-stage-MgPXZ9/spack-build/./gcc/xgcc: No such file or directory
make[3]: *** [cshift1_4.lo] Error 1
make[3]: *** Waiting for unfinished jobs....
```

I haven't been building `gcc` myself for long time....

What platform are you on? For `5.2.0`, try building `~binutils`. For `4.9.4`, it looks like the patch I added broke things. It works for older versions, so they must've backported it. I'll submit a PR shortly.


Resolving patch:
diff --git a/var/spack/repos/builtin/packages/gcc/package.py b/var/spack/repos/builtin/packages/gcc/package.py
--- a/var/spack/repos/builtin/packages/gcc/package.py
+++ b/var/spack/repos/builtin/packages/gcc/package.py
@@ -58,7 +58,7 @@ class Gcc(Package):
         provides('golang', when='@4.7.1:')
 
     patch('piclibs.patch', when='+piclibs')
-    patch('gcc-backport.patch', when='@4.7:5.3')
+    patch('gcc-backport.patch', when='@4.7:4.9.2,5:5.3')
 
     def install(self, spec, prefix):
         # libjava/configure needs a minor fix to install into spack paths.


Test patch:
Repository: spack/spack

Issue:
`spack test`: ~5x slowdown in tests involving `MockDatabase`?
Information on where I am working:
```console
$ uname -a 
Linux nuvolari 4.4.0-45-generic #66~14.04.1-Ubuntu SMP Wed Oct 19 15:05:38 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
```
If I run any test involving `MockDatabase` from the latest develop:
```console
$ git checkout develop
Already on 'develop'
Your branch is up-to-date with 'origin/develop'.

$ git branch -v
* develop a260e7e Bugfix in concretize_compiler (#2196)

$ spack test cmd.uninstall
==> Installing mpileaks
==> Installing mpich
==> Building mpich [Package]
...
----------------------------------------------------------------------
Ran 1 test in 40.622s

OK
==> Tests Complete.
      1 tests run
      0 failures
      0 errors
==> OK
```
Going back over the last commits, the first one at which I get the old performance is:
```console
$ git checkout 4e6d535
Note: checking out '4e6d535'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b new_branch_name

HEAD is now at 4e6d535... Set Clang as the default compiler on macOS (#2225)

$ spack test cmd.uninstall
==> Installing mpileaks
==> Installing mpich
==> Building mpich [Package]
...
----------------------------------------------------------------------
Ran 1 test in 7.855s

OK
==> Tests Complete.
      1 tests run
      0 failures
      0 errors
==> OK
```
This seems to point to #1822 as the source of this slowdown. Can anybody else reproduce the issue ?


Discussion:

Resolving patch:
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -703,12 +703,7 @@ def _make_root_stage(self, fetcher):
         # Construct a path where the stage should build..
         s = self.spec
         stage_name = "%s-%s-%s" % (s.name, s.version, s.dag_hash())
-
-        # Check list_url alternative archive URLs
-        dynamic_fetcher = fs.from_list_url(self)
-        alternate_fetchers = [dynamic_fetcher] if dynamic_fetcher else None
-        stage = Stage(fetcher, mirror_path=mp, name=stage_name, path=self.path,
-                      alternate_fetchers=alternate_fetchers)
+        stage = Stage(fetcher, mirror_path=mp, name=stage_name, path=self.path)
         return stage
 
     def _make_stage(self):


Test patch: