Repository: spack/spack

Issue:
Command Line Defaults
This PR adds command line defaults to Spack.  Default command line arguments can now be specified in a configuration file called `aliases.yaml`.  For example:
```
aliases:
    all: --debug
    spec: --install-status --long
    install: --install-status
    setup: --default-version
```

Example: When the user types `spack spec xz`, that will be converted to `spack --debug spec --install-status --long xz`.

Question: What should the YAML file be called?
 * `aliases.yaml` (because it's kind of like Bash aliases)? 
 * `defaults.yaml` (because it sets defaults for command lines)?
 * `default-args.yaml` (because that's more specific than `defaults.yaml`)?



Discussion:
I like `aliases.yaml`.

If I understand correctly, with this PR I'll be able to tell `spack install` that it should always be `--verbose`, right?
> If I understand correctly, with this PR I'll be able to tell `spack install` that it should always be `--verbose`, right?

Exactly.  Or for machines that require `--dirty`, an `aliases.yaml` file can be put in the appropriate arch-specific configuration adding the `--dirty` flag to `spack install`.

All the tests pass, except the doc test.  I'm getting the following error from Travis:
```
Warning, treated as error:
WARNING: Unexpected return code 2 from command u'spack info mpich'
```
When I try it myself, I get this behavior too:
```
$ spack info mpich
usage: spack info [-h] PACKAGE
spack info: error: too few arguments
```
Looks like this PR still needs a little work...
Maybe you can't parse args multiple times? It seemed a little hacky anyway.
Also, I can't believe the documentation tests are the only thing that catches such a grave bug.

> Maybe you can't parse args multiple times?

Good guess, but actually you can.

> It seemed a little hacky anyway.

 “Life is pain, highness. Anyone who says differently is selling something.”
― William Goldman

This PR is too good to throw away (IMHO), so I just re-did my `argparse` Jujitsu.  The idea now is to find the position of the subcommand in the command line, and split up the original arguments around it.  Then re-assemble the command line with the defaults added in.  I verified that Spack works just fine with repeated flags, in case the user specified a flag that was already in the defaults.

I find the position of the subcommand by first parsing the command line with a parser that has just the args for main Spack, leaving everything from the Spack command on as unparsed.  Then I use list lengths to split up the original args.

> Also, I can't believe the documentation tests are the only thing that catches such a grave bug.

I'm hoping I don't have to find a solution to that problem in this PR.
I like the concept of simplifying the spack command line, but I have reservations about this for the same reason the [git maintainers do](http://git.661346.n2.nabble.com/allowing-aliases-to-override-builtins-to-support-default-options-td2438491.html).  Specifically:

> > Currently git does not allow aliases to override builtins. I 
> > understand the reasoning behind this, but I wonder if it's overly 
> > conservative. 
> 
> It is not. 
> 
> > Most shells support overriding commands with aliases, and I'm not sure 
> > why git needs to be more conservative than the shell. 
> 
> Because sane shells do not expand aliases when used in a script, and gives 
> a handy way to defeat the alias even from the command line. 
> 
>     $ alias ls='ls -aF' 
>     $ echo ls >script 
>     $ chmod +x script 
> 
> and compare: 
> 
>     $ ./script 
>     $ ls 
>     $ /bin/ls 

I tend to agree with the issues raised above that if you allow everything in this PR, then it's going to be *very* hard to write a script for Spack that works reproducibly for different users.  I would like to keep Spack scriptable and to ensure a little bit that when we talk to a user on GitHub, that we're talking about the same semantics for a given command.

Also, the settings here overlap with some of those in `config.yaml`.  For example:

> Exactly. Or for machines that require --dirty, an aliases.yaml file can be put in the appropriate arch-specific configuration adding the --dirty flag to spack install.

Why not just set `dirty: true` in `config.yaml`?  That exists.  `color: true` may soon exist (see #2441).  We have control over what defaults we allow to be set here (i.e., we can keep them sane), and I don't mind adding more settings in `config.yaml`.  

That said, I would welcome a PR that allows [real aliases](https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases), similar to git's.  e.g.:

```yaml
aliases:
    dinstall: install --dirty
    in: install
    rm: uninstall
```

The one caveat would be the same as the one `git` enforces: aliases cannot override builtin commands.  That has the advantage, I think, of providing all the power you want from *this* PR, but not trampling the scripting capabilities.  Your options would have to be implemented as separate commands, but that's a minor thing and I think it will serve to keep future conversations we have on github more sane.

The other cool thing about `git` aliases is that they can take arguments.  I don't think that would actually be hard to implement in Spack, and people could likely implement even cooler things with that capability.

So basically I am saying I like this, but I think we should take a page from `git` and apply the same caution that works for them.  I think the time spent improving this will be worth it.

Thoughts?
> I tend to agree with the issues raised above that if you allow everything in this PR, then it's going to be *very* hard to write a script for Spack that works reproducibly for different users.  I would like to keep Spack scriptable and to ensure a little bit that when we talk to a user on GitHub, that we're talking about the same semantics for a given command.

Let's look at the situation more holistically.  Spack scripts are already non-reproducible between users, due to the large number of things that can ALREADY be in `packages.yaml`.  Having `spack install` rely on what's already installed (a common request, and one I believe is "on the docket") would make Spack even more non-reproducible, even for a *single* user.

Luckily, it is possible to use Spack in reproducible way.  With #2686 in place, I have completely emptied my `~/.spack/packages.yaml` file, splitting it into 4 separate `packages.yaml` files in 4 scopes: (a) CentOS7-specific, (b) Project-specific, (c) Specific to certain variants I use in some builds of my project, (d) Sets appropriate versions to `develop`.  I now specify the Spack configuration scopes I want to use on the command line, where it becomes explicit and reproducible.  I have two Bash aliases I use for this:
```
# Use to install released software
alias rspack='\spack -c ~/spscopes/centos7 -c ~/spscopes/gissversions'
# Use to build software in development
alias dspack='\spack -c ~/spscopes/centos7 -c ~/spscopes/gissversions -c ~/spscopes/twoway -c ~/spscopes/develop'
# Disable raw Spack in case I forget
alias spack='nothing'
```

By moving configuration from the user-level scope to command line scopes, I have made things reproducible where they were not before.

I could do the same thing with my `aliases.yaml` file: put them in a scope that I specify on the command line, and then use Bash aliases to specify that scope.  One could even enforce that stipulating that `aliases.yaml` *must* come from a scope specified on the command line; however, why should Spack enforce this for `aliases.yaml` but not `packages.yaml` --- both of which can cause severe reproducibility problems if not managed carefully.

Finally, let's look at what I'm actually *using* my `aliases.yaml` for (which I do have in `~/.spack/aliases.yaml`):
```
aliases:
    # all: --debug
    spec: --install-status --long
    install: --install-status
    #setup: --default-version
    setup: --install-status
    info:
```
As you can see, *none* of the alias commands I've selected have any effect on reproducibility.

**Conclusion:** Reproducibility in Spack is a significant issue, and user can shoot themselves in the foot.  This PR does not particularly increase or decrease that possibility.  However, #2686 DOES provide a tool that allows users to work in a reproducible way, if they so choose.

> > Exactly. Or for machines that require --dirty, an aliases.yaml file can be put in the appropriate arch-specific configuration adding the --dirty flag to spack install.
> 
> Why not just set `dirty: true` in `config.yaml`?  That exists.  `color: true` may soon exist (see #2441).  We have control over what defaults we allow to be set here (i.e., we can keep them sane), and I don't mind adding more settings in `config.yaml`.  

I suppose that's another way to go about it, and would be appropriate for the aliases I wanted this PR for.  However... this PR was quick, and in one fell swoop adds all possibility functionality --- without changing the code that uses those configuration options.  Adding the parameters one by one to `config.yaml` would be a lot more work.  And unless we have a systematic approach to allowing certain options to be set both on the command line or in `config.yaml`, the two will get out of sync again as somebody adds a new command line options.

> That said, I would welcome a PR that allows [real aliases](https://git-scm.com/book/en/v2/Git-Basics-Git-Aliases), similar to git's.  e.g.:
> ```yaml
> aliases:
>     dinstall: install --dirty
>     in: install
>     rm: uninstall
> ```

That doesn't cover the use case I want this PR for.  Now users have to type `spack dinstall` instead of `spack install` (more on that below).

Nor is your example convincing.  Because typically `--dirty` is used on certain systems that need it (such as our supercomputer with its compiler built in a crazy brain-dead way).  In that case, you need to *always* type `spack install --dirty`, not `spack install`.  The dangers is people will forget, and they'll type `spack install`.  Making them type `spack dinstall` instead of `spack install` saves a few keystrokes, but it doesn't really help that core problem.

> That has the advantage, I think, of providing all the power you want from *this* PR,

It provides none of the power I want from this PR.  If I'm willing to type non-standard Spack commands (eg `spack dinstall` instad of `spack install`), I might as well just do it in Bash:
```
alias spack-spec='spack spec --install-status --long'
alias spack-install='spack install --install-status'
alias spack-setup='spack setup --install-status'
```

This would be minorly inconvenient for me, but I might get used to it.  However, it could be embarassing in other cases.  Suppose I'm on a system using some crazy compiler that requires `--dirty`.  So I do:
```
alias spack-diy='spack diy --dirty'
alias spack-install='spack install --dirty'
```
Now I sit my users down with a Spack tutorial and tell them to type `spack-install` every time they see `spack install`; but to not similarly modify the other 50 Spack commands they might type.  You can see how well THAT will go over.  No... if my *system* requires `--dirty`, the most effective thing to do will be to configure Spack that way in a System-wide scope.

> but not trampling the scripting capabilities.

As long as you only put `aliases.yaml` (and `packages.yaml` too) in scopes specified on the command line, scripting capabilities will be unaffected.

> The other cool thing about `git` aliases is that they can take arguments.  I don't think that would actually be hard to implement in Spack, and people could likely implement even cooler things with that capability.

Again... not the functionality I'm looking for in this PR.

> Thoughts?

The proposal to add appropriate options to `config.yaml` is the only one that provides the functionaly I needed out of this PR.  Unless someone wants to do that work, I think this PR is the best we have for now, and it gets the job done.

I would definitely like the ability to set default flags for Spack subcommands. I almost always want:
```
$ spack install --verbose
$ spack spec --install-status
$ spack find --paths --variants
```
I don't particularly care whether this is done by adding an `aliases.yaml` or adding these flags to `config.yaml`. But I agree with @citibeth that `config.yaml` support may not be updated as quickly as new flags are added or old flags are removed/renamed. I don't see a major issue with reproducibility. Yes, Spack should be scriptable, and yes, if someone aliases `spack install` to `spack install --only dependencies`, then Spack is no longer scriptable. What if we add a flag to Spack like `spack --no-aliases` that ignores all user provided aliases? This makes it still scriptable.
Do others have an opinion on this?  @alalazo? @davydden? @mamelara @mplegendre? @becker33 ? @lee218llnl ? sounds like I might be in the minority.
Personal minor concern, mostly OT: I would not grow the array of configuration files. If there is already `config.yaml`, maybe the functionality should be added through it. Otherwise in a year one may end up having tens of files and it would be a pain to remember which one actually influence what.
I'm wondering why we have separate YAML files to begin with.  They could
all be concatenated into one.

On Tue, Jan 10, 2017 at 10:55 AM, Denis Davydov <notifications@github.com>
wrote:

> Personal minor concern, mostly OT: I would not grow the array of
> configuration files. If there is already config.yaml, maybe the
> functionality should be added through it. Otherwise in a year one may end
> up having tens of files and it would be a pain to remember which one
> actually influence what.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LLNL/spack/pull/2705#issuecomment-271613623>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cd92qwXAdRAzSZoLozdD4ZKo45AAWks5rQ6oPgaJpZM4LYn1r>
> .
>

@tgamblin I tend to agree with your opinion above, in particular:

> Also, the settings here overlap with some of those in config.yaml. 
> Why not just set dirty: true in config.yaml? That exists. color: true may soon exist (see #2441). We have control over what defaults we allow to be set here (i.e., we can keep them sane), and I don't mind adding more settings in config.yaml.


 and with @davydden on the number of configuration files.


> and I don't mind adding more settings in config.yaml.

Looking at the specific command-line options that @adamjstewart and I have
said we want to use this for... do you mind adding *those* to *config.yaml*?


On Tue, Jan 10, 2017 at 11:14 AM, Massimiliano Culpo <
notifications@github.com> wrote:

> @tgamblin <https://github.com/tgamblin> I tend to agree with your opinion
> above, in particular:
>
> Also, the settings here overlap with some of those in config.yaml.
> Why not just set dirty: true in config.yaml? That exists. color: true may
> soon exist (see #2441 <https://github.com/LLNL/spack/issues/2441>). We
> have control over what defaults we allow to be set here (i.e., we can keep
> them sane), and I don't mind adding more settings in config.yaml.
>
> and with @davydden <https://github.com/davydden> on the number of
> configuration files.
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/LLNL/spack/pull/2705#issuecomment-271619335>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AB1cdxBGlrNxkYF8YzUE_MHmHwiiJNpDks5rQ659gaJpZM4LYn1r>
> .
>

> I'm wondering why we have separate YAML files to begin with. They could all be concatenated into one.

Just because they _could_ doesn't mean they _should_. They provide logical separation, so if I ask you for your compiler settings, you can give me your `compilers.yaml` and not a several hundred line `behemoth.yaml`. Or if I want to see your package settings, you can just give me your `packages.yaml`.

I don't mind a new `aliases.yaml`. I also don't mind putting them in `config.yaml` either.
> Looking at the specific command-line options that @adamjstewart and I have
said we want to use this for... do you mind adding *those* to *config.yaml*?

Nope.  I like those actually.  I also think making `spack install` show the `spack spec` output (or some type of summary) by default is a good idea.
RE: configuration proliferation: I also like fewer config files, though I am not sure I would like a single one.  So far:

1. `config.yaml`: I'd like to put most new things here.
1. `packages.yaml`: rationale was to make this separate because it can get very long.  It's per-package data so I think it deserves its own file.
1. `compilers.yaml`: again, this is long, per-compiler data. At LLNL has like 30 elements when `spack` is first run.  `compilers.yaml` should eventually be consolidated with `packages.yaml`, when compilers are normal, potentially external dependencies.  So this one will go away.
1. `mirrors.yaml`, `repos.yaml`: these could probably be consolidated with `config.yaml`

One near-term concern I have about concatenating is that commands edit the YAML files, and while I've hacked the YAML parser a bit, we haven't hacked it to round-trip comments.  So you'll lose comments for things like `spack compiler add`.  If you consolidate the files, you'd lose any comments in your larger `config.yaml` too.

That said, consolidating is easy.  The files all start with unique section keys (this was intentional) so concatenating them doesn't actually make them harder to parse or differentiate.  @mplegendre and i have talked about allowing the top-level sections to be either separate files or concatenated into one. I think that would be easy to pull off.

Could an argument `--no-aliases` be added to resolve this issue @tgamblin @citibeth? Personally I think @citibeth is probably right and there isn't really a problem adding this functionality, but if there was a special argument to turn it off that should relieve most of @tgamblin's concerns. Probably easier said than done though..
@mplegendre  To be more specific... I would like the *option* of providing configuration scopes as a consolidated single file, rather than a directory of YAML files.  I don't necessarily want to do this to my `~/.spack` directory.  But I *am* now generating a bunch of user-defined configuration scopes that go on top of the standard scopes (see #2686).  For those, I would like to create a single file per scope, rather than a directory full of files.  I'm not bothered by auto-generation issues, I don't intend to auto-generate these configurations anyway.

As we formalize ideas around Spack environments, I think it's worthwhile separating options that change how concretization works into a separate file.  A quick check through docs on `config.yaml` indicates that is likely already the case (only `packages.yaml` changes concretization).
> Personally I think @citibeth is probably right and there isn't really a problem adding this functionality

I said more than that.  Actually, I said that Spack is *already* not scriptable because of `packages.yaml`.  That horse has already left the barn.

The article on `git` explained why the `--no-aliases` option isn't preferable.  But at least for the options I want to default, there is no need for it anyway.
Superseded by #17229 

Resolving patch:
diff --git a/lib/spack/spack/cmd/commands.py b/lib/spack/spack/cmd/commands.py
--- a/lib/spack/spack/cmd/commands.py
+++ b/lib/spack/spack/cmd/commands.py
@@ -796,7 +796,9 @@ def names(args: Namespace, out: IO) -> None:
     commands = copy.copy(spack.cmd.all_commands())
 
     if args.aliases:
-        commands.extend(spack.main.aliases.keys())
+        aliases = spack.config.get("config:aliases")
+        if aliases:
+            commands.extend(aliases.keys())
 
     colify(commands, output=out)
 
@@ -812,8 +814,10 @@ def bash(args: Namespace, out: IO) -> None:
     parser = spack.main.make_argument_parser()
     spack.main.add_all_commands(parser)
 
-    aliases = ";".join(f"{key}:{val}" for key, val in spack.main.aliases.items())
-    out.write(f'SPACK_ALIASES="{aliases}"\n\n')
+    aliases_config = spack.config.get("config:aliases")
+    if aliases_config:
+        aliases = ";".join(f"{key}:{val}" for key, val in aliases_config.items())
+        out.write(f'SPACK_ALIASES="{aliases}"\n\n')
 
     writer = BashCompletionWriter(parser.prog, out, args.aliases)
     writer.write(parser)
diff --git a/lib/spack/spack/main.py b/lib/spack/spack/main.py
--- a/lib/spack/spack/main.py
+++ b/lib/spack/spack/main.py
@@ -16,11 +16,13 @@
 import os.path
 import pstats
 import re
+import shlex
 import signal
 import subprocess as sp
 import sys
 import traceback
 import warnings
+from typing import List, Tuple
 
 import archspec.cpu
 
@@ -49,9 +51,6 @@
 #: names of profile statistics
 stat_names = pstats.Stats.sort_arg_dict_default
 
-#: top-level aliases for Spack commands
-aliases = {"concretise": "concretize", "containerise": "containerize", "rm": "remove"}
-
 #: help levels in order of detail (i.e., number of commands shown)
 levels = ["short", "long"]
 
@@ -359,7 +358,10 @@ def add_command(self, cmd_name):
             module = spack.cmd.get_module(cmd_name)
 
             # build a list of aliases
-            alias_list = [k for k, v in aliases.items() if v == cmd_name]
+            alias_list = []
+            aliases = spack.config.get("config:aliases")
+            if aliases:
+                alias_list = [k for k, v in aliases.items() if shlex.split(v)[0] == cmd_name]
 
             subparser = self.subparsers.add_parser(
                 cmd_name,
@@ -670,7 +672,6 @@ def __init__(self, command_name, subprocess=False):
                 Windows, where it is always False.
         """
         self.parser = make_argument_parser()
-        self.command = self.parser.add_command(command_name)
         self.command_name = command_name
         # TODO: figure out how to support this on windows
         self.subprocess = subprocess if sys.platform != "win32" else False
@@ -702,13 +703,14 @@ def __call__(self, *argv, **kwargs):
 
         if self.subprocess:
             p = sp.Popen(
-                [spack.paths.spack_script, self.command_name] + prepend + list(argv),
+                [spack.paths.spack_script] + prepend + [self.command_name] + list(argv),
                 stdout=sp.PIPE,
                 stderr=sp.STDOUT,
             )
             out, self.returncode = p.communicate()
             out = out.decode()
         else:
+            command = self.parser.add_command(self.command_name)
             args, unknown = self.parser.parse_known_args(
                 prepend + [self.command_name] + list(argv)
             )
@@ -716,7 +718,7 @@ def __call__(self, *argv, **kwargs):
             out = io.StringIO()
             try:
                 with log_output(out, echo=True):
-                    self.returncode = _invoke_command(self.command, self.parser, args, unknown)
+                    self.returncode = _invoke_command(command, self.parser, args, unknown)
 
             except SystemExit as e:
                 self.returncode = e.code
@@ -870,6 +872,46 @@ def restore_macos_dyld_vars():
             os.environ[dyld_var] = os.environ[stored_var_name]
 
 
+def resolve_alias(cmd_name: str, cmd: List[str]) -> Tuple[str, List[str]]:
+    """Resolves aliases in the given command.
+
+    Args:
+        cmd_name: command name.
+        cmd: command line arguments.
+
+    Returns:
+        new command name and arguments.
+    """
+    all_commands = spack.cmd.all_commands()
+    aliases = spack.config.get("config:aliases")
+
+    if aliases:
+        for key, value in aliases.items():
+            if " " in key:
+                tty.warn(
+                    f"Alias '{key}' (mapping to '{value}') contains a space"
+                    ", which is not supported."
+                )
+            if key in all_commands:
+                tty.warn(
+                    f"Alias '{key}' (mapping to '{value}') attempts to override"
+                    " built-in command."
+                )
+
+    if cmd_name not in all_commands:
+        alias = None
+
+        if aliases:
+            alias = aliases.get(cmd_name)
+
+        if alias is not None:
+            alias_parts = shlex.split(alias)
+            cmd_name = alias_parts[0]
+            cmd = alias_parts + cmd[1:]
+
+    return cmd_name, cmd
+
+
 def _main(argv=None):
     """Logic for the main entry point for the Spack command.
 
@@ -962,7 +1004,7 @@ def _main(argv=None):
 
     # Try to load the particular command the caller asked for.
     cmd_name = args.command[0]
-    cmd_name = aliases.get(cmd_name, cmd_name)
+    cmd_name, args.command = resolve_alias(cmd_name, args.command)
 
     # set up a bootstrap context, if asked.
     # bootstrap context needs to include parsing the command, b/c things
@@ -974,14 +1016,14 @@ def _main(argv=None):
         bootstrap_context = bootstrap.ensure_bootstrap_configuration()
 
     with bootstrap_context:
-        return finish_parse_and_run(parser, cmd_name, env_format_error)
+        return finish_parse_and_run(parser, cmd_name, args.command, env_format_error)
 
 
-def finish_parse_and_run(parser, cmd_name, env_format_error):
+def finish_parse_and_run(parser, cmd_name, cmd, env_format_error):
     """Finish parsing after we know the command to run."""
     # add the found command to the parser and re-run then re-parse
     command = parser.add_command(cmd_name)
-    args, unknown = parser.parse_known_args()
+    args, unknown = parser.parse_known_args(cmd)
 
     # Now that we know what command this is and what its args are, determine
     # whether we can continue with a bad environment and raise if not.
diff --git a/lib/spack/spack/schema/config.py b/lib/spack/spack/schema/config.py
--- a/lib/spack/spack/schema/config.py
+++ b/lib/spack/spack/schema/config.py
@@ -92,6 +92,7 @@
             "url_fetch_method": {"type": "string", "enum": ["urllib", "curl"]},
             "additional_external_search_paths": {"type": "array", "items": {"type": "string"}},
             "binary_index_ttl": {"type": "integer", "minimum": 0},
+            "aliases": {"type": "object", "patternProperties": {r"\w[\w-]*": {"type": "string"}}},
         },
         "deprecatedProperties": {
             "properties": ["terminal_title"],


Test patch:
diff --git a/lib/spack/spack/test/cmd/commands.py b/lib/spack/spack/test/cmd/commands.py
--- a/lib/spack/spack/test/cmd/commands.py
+++ b/lib/spack/spack/test/cmd/commands.py
@@ -58,6 +58,24 @@ def test_subcommands():
     assert "spack compiler add" in out2
 
 
+@pytest.mark.not_on_windows("subprocess not supported on Windows")
+def test_override_alias():
+    """Test that spack commands cannot be overriden by aliases."""
+
+    install = spack.main.SpackCommand("install", subprocess=True)
+    instal = spack.main.SpackCommand("instal", subprocess=True)
+
+    out = install(fail_on_error=False, global_args=["-c", "config:aliases:install:find"])
+    assert "install requires a package argument or active environment" in out
+    assert "Alias 'install' (mapping to 'find') attempts to override built-in command" in out
+
+    out = install(fail_on_error=False, global_args=["-c", "config:aliases:foo bar:find"])
+    assert "Alias 'foo bar' (mapping to 'find') contains a space, which is not supported" in out
+
+    out = instal(fail_on_error=False, global_args=["-c", "config:aliases:instal:find"])
+    assert "install requires a package argument or active environment" not in out
+
+
 def test_rst():
     """Do some simple sanity checks of the rst writer."""
     out1 = commands("--format=rst")