Repository: spack/spack

Issue:
Add global 'source' variant to install source code
Sometimes it would be useful to have the source code together with the build, so I would like to add a variant (`+source`) in somewhere in the package base class (`spack/spack/package.py`) to trigger a copy of the source code (after patching, before configure/cmake/build) to `<prefix>/src`.

Any suggestions?


Discussion:
@mplegendre and I have talked about this a bit before -- we'd like to keep the source around for debugging.

Some questions:
1. Were you planning to just keep the source tarball around, or do you want an expanded source directory?
2. Do we need a global variant, which modifies the hash for no reason related to the artifact, or would it make sense to just make this an install option and put the source in the `$prefix/.spack/` directory?  If you did the latter you could always add the source to already-installed things, as it's not an interface guarantee in the spec.
3. What about debug information?  This is the part we never fully hashed out.
    1. Ideally, you'd build the binary against the already installed source, so the debug info would already point to the right place.  (1) only really works well for out-of-source builds, as otherwise you end up with a bunch of .o files and things in the source directory.  That, or it forces you to do the build on a potentially slow shared filesystem instead of `$TMP`.
    1. The other option would be to do the build as usual and patch the debug info on install.  This would be my preference -- but I don't think there is currently a similarly robust analog to `patchelf` for DWARF.

Thoughts?
> add a variant (+source)

most certainly should not be a variant as this does NOT change the functionality or behaviour of the installed package. It's more like a `--run-tests` flag, which does some extra steps. 

Btw, there was a discussion somewhere on debugging and keeping the source or patching the installed libs so that you can run though source during debugging. I could not find it, however....
@tgamblin damn, you wrote about debugging just 41 seconds before myself :smile:
Is there already precedent for "global" variants? If so...didn't know that and I have always been in the market for a few...

- debug vs. optimized builds
  - and can't a debug build imply the meaning of the proposed 'source' variant here?
- pic vs. no-pic
- 32 vs. 64 bit indexing (used in many HPC solver libs, I/O libs, etc.)
- static vs. dynamic libs (or dynamic and static vs. static only)
- api language bindings (for libs) c, cxx, fort
- required, common-optional, all dependent libs
  - required is for required dependencies (e.g UNconditional depends_on)
  - common-optional are dependencies that are optional but frequently used (e.g. zlib)
  - all is the whole kitchen sink of dependencies
  - of course, having per-dependency variant controls is probably still useful but a global mechanism for trimming dependency tree would be extremely useful IMHO.


> Were you planning to just keep the source tarball around, or do you want an expanded source directory?

I would like the expanded source with all patches applied.

> Do we need a global variant, which modifies the hash for no reason related to the artifact, or would it make sense to just make this an install option and put the source in the $prefix/.spack/ directory? If you did the latter you could always add the source to already-installed things, as it's not an interface guarantee in the spec.

I would like to use `depends_on(package+source)` as a mechanism to distribute binaries incl. their sources to hardware vendors. (/cc @richards12).


Resolving patch:
diff --git a/lib/spack/spack/cmd/install.py b/lib/spack/spack/cmd/install.py
--- a/lib/spack/spack/cmd/install.py
+++ b/lib/spack/spack/cmd/install.py
@@ -68,6 +68,9 @@ def setup_parser(subparser):
     subparser.add_argument(
         '--restage', action='store_true', dest='restage',
         help="if a partial install is detected, delete prior state")
+    subparser.add_argument(
+        '--source', action='store_true', dest='install_source',
+        help="install source files in prefix")
     subparser.add_argument(
         '-n', '--no-checksum', action='store_true', dest='no_checksum',
         help="do not check packages against checksum")
@@ -314,6 +317,7 @@ def install(parser, args, **kwargs):
         'keep_prefix': args.keep_prefix,
         'keep_stage': args.keep_stage,
         'restage': args.restage,
+        'install_source': args.install_source,
         'install_deps': 'dependencies' in args.things_to_install,
         'make_jobs': args.jobs,
         'run_tests': args.run_tests,
diff --git a/lib/spack/spack/package.py b/lib/spack/spack/package.py
--- a/lib/spack/spack/package.py
+++ b/lib/spack/spack/package.py
@@ -1193,6 +1193,7 @@ def _update_explicit_entry_in_db(self, rec, explicit):
     def do_install(self,
                    keep_prefix=False,
                    keep_stage=False,
+                   install_source=False,
                    install_deps=True,
                    skip_patch=False,
                    verbose=False,
@@ -1213,6 +1214,8 @@ def do_install(self,
             keep_stage (bool): By default, stage is destroyed only if there
                 are no exceptions during build. Set to True to keep the stage
                 even with exceptions.
+            install_source (bool): By default, source is not installed, but
+                for debugging it might be useful to keep it around.
             install_deps (bool): Install dependencies before installing this
                 package
             skip_patch (bool): Skip patch stage of build if True.
@@ -1260,6 +1263,7 @@ def do_install(self,
                 dep.package.do_install(
                     keep_prefix=keep_prefix,
                     keep_stage=keep_stage,
+                    install_source=install_source,
                     install_deps=install_deps,
                     fake=fake,
                     skip_patch=skip_patch,
@@ -1312,6 +1316,13 @@ def build_process():
                 if fake:
                     self.do_fake_install()
                 else:
+                    source_path = self.stage.source_path
+                    if install_source and os.path.isdir(source_path):
+                        src_target = join_path(
+                            self.spec.prefix, 'share', self.name, 'src')
+                        tty.msg('Copying source to {0}'.format(src_target))
+                        install_tree(self.stage.source_path, src_target)
+
                     # Do the real install in the source directory.
                     self.stage.chdir_to_source()
 


Test patch:
diff --git a/lib/spack/spack/test/cmd/install.py b/lib/spack/spack/test/cmd/install.py
--- a/lib/spack/spack/test/cmd/install.py
+++ b/lib/spack/spack/test/cmd/install.py
@@ -24,6 +24,7 @@
 ##############################################################################
 import argparse
 import os
+import filecmp
 
 import pytest
 
@@ -130,3 +131,14 @@ def test_install_output_on_python_error(builtin_mock, mock_archive, mock_fetch,
     assert isinstance(install.error, spack.build_environment.ChildError)
     assert install.error.name == 'InstallError'
     assert 'raise InstallError("Expected failure.")' in out
+
+
+def test_install_with_source(
+        builtin_mock, mock_archive, mock_fetch, config, install_mockery):
+    """Verify that source has been copied into place."""
+    install('--source', '--keep-stage', 'trivial-install-test-package')
+    spec = Spec('trivial-install-test-package').concretized()
+    src = os.path.join(
+        spec.prefix.share, 'trivial-install-test-package', 'src')
+    assert filecmp.cmp(os.path.join(mock_archive.path, 'configure'),
+                       os.path.join(src, 'configure'))