Repository: spack/spack

Issue:
Requiring a specific name for the envfile breaks existing workflows
### Steps to reproduce

We generate and store multiple similar `spack.yaml` files from a template, such as:
```console
critic@2.12.02-clang@14.0.6-cxx17-prof-gcc@4.8.5.yaml
critic@2.12.02-clang@15.0.7-cxx17-prof-gcc@4.8.5.yaml
critic@2.12.02-gcc@12.2.0-cxx17-prof-gcc@4.8.5.yaml
critic@2.12.02-gcc@12.2.0-cxx20-prof-gcc@4.8.5.yaml
critic@2.12.02-gcc@13.1.0-cxx20-prof-gcc@4.8.5.yaml
critic@2.12.02-gcc@8.2.0-cxx17-prof-gcc@4.8.5.yaml
critic@2.12.02-gcc@9.3.0-cxx17-prof-gcc@4.8.5.yaml
```
We then use `spack env create <env-name> <file>` to create a suitable environment and then concretize and build it.

We were also planning to extend this to use concretized environment config files, which would have been similarly named.

#36927 breaks this by requiring that the provided environment config file actually be named `spack.yaml` or `spack.lock`.

Is there really no other way to distinguish one type of environment config file from the other without constraining the name?

### Error message

_No response_

### Information on your system

* **Spack:** 0.20.0.dev0
* **Python:** 3.6.8
* **Platform:** linux-scientific7-skylake_avx512
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
I had a similar discussion on this yesterday with @mdorier on slack [here](https://spackpm.slack.com/archives/CMHK7MF51/p1683023649635819) Posting also a screenshot, since the discussion was public and it will be lost in time:

![Screenshot from 2023-05-03 20-47-40](https://user-images.githubusercontent.com/4199709/236014411-796cbab0-d8a0-4c97-aecb-41e8e86a6837.png)

---

The good news is that the lines enforcing this behavior are much localized:

https://github.com/spack/spack/blob/59e2ef6ad64b12fd7f168308f3ac87b7472d2994/lib/spack/spack/environment/environment.py#L2536-L2541

I remain of the opinion expressed above, but in case it wouldn't be hard to lift enforcing the filename.
For the record, yesterday I also checked:
- https://spack-tutorial.readthedocs.io/en/latest/tutorial_environments.html# 
- https://spack.readthedocs.io/en/latest/environments.html

and couldn't find a single example where we use a name that is not `spack.yaml`. Before #36927, anything was fine as a name when creating an environment, but when activating the environment a `spack.yaml` file was required. 

For the lockfile, after #36927 it should be named `spack.lock`, before `*.lock` (which was also inconsistent and not documented).
I always interpreted the 'spack.yaml' or 'spack.lock' as describing the character and purpose of the file rather than its name.
The ability to generate environment config files and store them for later use is quite important to us, and enforcing a single filename for each type of file and relying on (say) file location for specificity is a high risk for information loss. Just a few days ago on Slack maintainers channel @becker33 viewed favorably the idea of being able to accept environment config files via URL.

I would much prefer the behavior of assuming that a given file or URL describes a `spack.yaml`-type environment configuration unless one provides a non-default flag (say, `--concrete`). Unless I'm misreading things, since the given file was copied into place in the defined (or defaulted) environment directory as `spack.yaml` or `spack.lock` anyway by `spack env create`, it would seem that the filename provided on the command line is irrelevant if there is another way to distinguish  a `spack.yaml`-type file from a `spack.lock`-type one.
I'm fine reverting the place where we enforce the file name, but what should distinguish a lockfile from a manifest? Only the suffix? Also, I'd like to document the behavior properly and be consistent with what the command line help and documentation say.

For the record, any name for the manifest was allowed before:

![Screenshot from 2023-05-02 14-15-04](https://user-images.githubusercontent.com/4199709/236020590-ca585a61-3da1-41f4-8143-1f1767f28e88.png)

and seemed to derive from using an `else` condition after a check on whether the filename ends with `.lock`.

It looks like the code uses `endswith(...)`, so maybe it's still possible to call the file `<whatever>.spack.yaml`? (I haven't tried)

Also note that valid extensions for yaml files are either .yaml or .yml, so it's a bit odd that even `spack.yml` would be refused.
Let me submit a PR to go back to the previous behavior, and enforce consistency by changing the command line help and docs.

Resolving patch:
diff --git a/lib/spack/spack/cmd/env.py b/lib/spack/spack/cmd/env.py
--- a/lib/spack/spack/cmd/env.py
+++ b/lib/spack/spack/cmd/env.py
@@ -283,7 +283,7 @@ def env_create_setup_parser(subparser):
         "envfile",
         nargs="?",
         default=None,
-        help="optional init file; can be spack.yaml or spack.lock",
+        help="either a lockfile (must end with '.json' or '.lock') or a manifest file.",
     )
 
 
@@ -292,7 +292,7 @@ def env_create(args):
         # Expand relative paths provided on the command line to the current working directory
         # This way we interpret `spack env create --with-view ./view --dir ./env` as
         # a view in $PWD/view, not $PWD/env/view. This is different from specifying a relative
-        # path in spack.yaml, which is resolved relative to the environment file.
+        # path in the manifest, which is resolved relative to the manifest file's location.
         with_view = os.path.abspath(args.with_view)
     elif args.without_view:
         with_view = False
@@ -317,7 +317,7 @@ def _env_create(name_or_path, *, init_file=None, dir=False, with_view=None, keep
     Arguments:
         name_or_path (str): name of the environment to create, or path to it
         init_file (str or file): optional initialization file -- can be
-            spack.yaml or spack.lock
+            a JSON lockfile (*.lock, *.json) or YAML manifest file
         dir (bool): if True, create an environment in a directory instead
             of a named environment
         keep_relative (bool): if True, develop paths are copied verbatim into
@@ -355,8 +355,7 @@ def env_remove(args):
     """Remove a *named* environment.
 
     This removes an environment managed by Spack. Directory environments
-    and `spack.yaml` files embedded in repositories should be removed
-    manually.
+    and manifests embedded in repositories should be removed manually.
     """
     read_envs = []
     for env_name in args.rm_env:
@@ -568,7 +567,7 @@ def env_revert(args):
     # Check that both the spack.yaml and the backup exist, the inform user
     # on what is going to happen and ask for confirmation
     if not os.path.exists(manifest_file):
-        msg = "cannot fine the manifest file of the environment [file={0}]"
+        msg = "cannot find the manifest file of the environment [file={0}]"
         tty.die(msg.format(manifest_file))
     if not os.path.exists(backup_file):
         msg = "cannot find the old manifest file to be restored [file={0}]"
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -281,9 +281,12 @@ def create(
     A managed environment is created in a root directory managed by this Spack instance, so that
     Spack can keep track of them.
 
+    Files with suffix ``.json`` or ``.lock`` are considered lockfiles. Files with any other name
+    are considered manifest files.
+
     Args:
         name: name of the managed environment
-        init_file: either a "spack.yaml" or a "spack.lock" file or None
+        init_file: either a lockfile, a manifest file, or None
         with_view: whether a view should be maintained for the environment. If the value is a
             string, it specifies the path to the view
         keep_relative: if True, develop paths are copied verbatim into the new environment file,
@@ -303,9 +306,12 @@ def create_in_dir(
 ) -> "Environment":
     """Create an environment in the directory passed as input and returns it.
 
+    Files with suffix ``.json`` or ``.lock`` are considered lockfiles. Files with any other name
+    are considered manifest files.
+
     Args:
         manifest_dir: directory where to create the environment.
-        init_file: either a "spack.yaml" or a "spack.lock" file or None
+        init_file: either a lockfile, a manifest file, or None
         with_view: whether a view should be maintained for the environment. If the value is a
             string, it specifies the path to the view
         keep_relative: if True, develop paths are copied verbatim into the new environment file,
@@ -2496,7 +2502,8 @@ def initialize_environment_dir(
 ) -> None:
     """Initialize an environment directory starting from an envfile.
 
-    The envfile can be either a "spack.yaml" manifest file, or a "spack.lock" file.
+    Files with suffix .json or .lock are considered lockfiles. Files with any other name
+    are considered manifest files.
 
     Args:
         environment_dir: directory where the environment should be placed
@@ -2533,21 +2540,18 @@ def _ensure_env_dir():
         msg = f"cannot initialize environment, {envfile} is not a valid file"
         raise SpackEnvironmentError(msg)
 
-    if not str(envfile).endswith(manifest_name) and not str(envfile).endswith(lockfile_name):
-        msg = (
-            f"cannot initialize environment from '{envfile}', either a '{manifest_name}'"
-            f" or a '{lockfile_name}' file is needed"
-        )
-        raise SpackEnvironmentError(msg)
-
     _ensure_env_dir()
 
     # When we have a lockfile we should copy that and produce a consistent default manifest
-    if str(envfile).endswith(lockfile_name):
+    if str(envfile).endswith(".lock") or str(envfile).endswith(".json"):
         shutil.copy(envfile, target_lockfile)
         # This constructor writes a spack.yaml which is consistent with the root
         # specs in the spack.lock
-        EnvironmentManifestFile.from_lockfile(environment_dir)
+        try:
+            EnvironmentManifestFile.from_lockfile(environment_dir)
+        except Exception as e:
+            msg = f"cannot initialize environment, '{environment_dir}' from lockfile"
+            raise SpackEnvironmentError(msg) from e
         return
 
     shutil.copy(envfile, target_manifest)


Test patch:
diff --git a/lib/spack/spack/test/env.py b/lib/spack/spack/test/env.py
--- a/lib/spack/spack/test/env.py
+++ b/lib/spack/spack/test/env.py
@@ -3,6 +3,7 @@
 #
 # SPDX-License-Identifier: (Apache-2.0 OR MIT)
 """Test environment internals without CLI"""
+import filecmp
 import os
 import pickle
 import sys
@@ -316,20 +317,13 @@ def test_cannot_initiliaze_if_dirname_exists_as_a_file(tmp_path):
         ev.create_in_dir(dir_name)
 
 
-def test_cannot_initiliaze_if_init_file_does_not_exist(tmp_path):
+def test_cannot_initialize_if_init_file_does_not_exist(tmp_path):
     """Tests that initializing an environment passing a non-existing init file raises an error."""
     init_file = tmp_path / ev.manifest_name
     with pytest.raises(ev.SpackEnvironmentError, match="cannot initialize"):
         ev.create_in_dir(tmp_path, init_file=init_file)
 
 
-def test_cannot_initialize_from_random_file(tmp_path):
-    init_file = tmp_path / "foo.txt"
-    init_file.touch()
-    with pytest.raises(ev.SpackEnvironmentError, match="cannot initialize"):
-        ev.create_in_dir(tmp_path, init_file=init_file)
-
-
 def test_environment_pickle(tmp_path):
     env1 = ev.create_in_dir(tmp_path)
     obj = pickle.dumps(env1)
@@ -419,3 +413,64 @@ def test_preserving_comments_when_adding_specs(
 
     content = spack_yaml.read_text()
     assert content == expected_yaml
+
+
+@pytest.mark.parametrize("filename", [ev.lockfile_name, "as9582g54.lock", "m3ia54s.json"])
+@pytest.mark.regression("37410")
+def test_initialize_from_lockfile(tmp_path, filename):
+    """Some users have workflows where they store multiple lockfiles in the
+    same directory, and pick one of them to create an environment depending
+    on external parameters e.g. while running CI jobs. This test ensures that
+    Spack can create environments from lockfiles that are not necessarily named
+    'spack.lock' and can thus coexist in the same directory.
+    """
+
+    init_file = tmp_path / filename
+    env_dir = tmp_path / "env_dir"
+    init_file.write_text('{ "roots": [] }\n')
+    ev.initialize_environment_dir(env_dir, init_file)
+
+    assert os.path.exists(env_dir / ev.lockfile_name)
+    assert filecmp.cmp(env_dir / ev.lockfile_name, init_file, shallow=False)
+
+
+def test_cannot_initialize_from_bad_lockfile(tmp_path):
+    """Test that we fail on an incorrectly constructed lockfile"""
+
+    init_file = tmp_path / ev.lockfile_name
+    env_dir = tmp_path / "env_dir"
+
+    init_file.write_text("Not a legal JSON file\n")
+
+    with pytest.raises(ev.SpackEnvironmentError, match="from lockfile"):
+        ev.initialize_environment_dir(env_dir, init_file)
+
+
+@pytest.mark.parametrize("filename", ["random.txt", "random.yaml", ev.manifest_name])
+@pytest.mark.regression("37410")
+def test_initialize_from_random_file_as_manifest(tmp_path, filename):
+    """Some users have workflows where they store multiple lockfiles in the
+    same directory, and pick one of them to create an environment depending
+    on external parameters e.g. while running CI jobs. This test ensures that
+    Spack can create environments from manifest that are not necessarily named
+    'spack.yaml' and can thus coexist in the same directory.
+    """
+
+    init_file = tmp_path / filename
+    env_dir = tmp_path / "env_dir"
+
+    init_file.write_text(
+        """\
+spack:
+  view: true
+  concretizer:
+    unify: true
+  specs: []
+"""
+    )
+
+    ev.create_in_dir(env_dir, init_file)
+
+    assert not os.path.exists(env_dir / ev.lockfile_name)
+    assert os.path.exists(env_dir / ev.manifest_name)
+    assert filecmp.cmp(env_dir / ev.manifest_name, init_file, shallow=False)