Repository: spack/spack

Issue:
spack remove fails in very simple case
### Steps to reproduce

Using
* spack@develop (9438cac219cc7ff127e03bc8a09dcb5d67124bb7 from `Thu Aug 10 22:58:05 2023 +0200`)

**Update:** `git bisect` says this stopped working @ 3c3a4c75776ece43c95df46908dea026ac2a9276 (CC @alalazo)

spack.yaml
```
spack:
  view: false

  concretizer:
    reuse: false
    unify: false

  compilers:
  - compiler:
      spec: gcc@=11.4.0
      paths:
        cc: /usr/bin/gcc
        cxx: /usr/bin/g++
        f77: /usr/bin/gfortran
        fc: /usr/bin/gfortran
      flags: {}
      operating_system: ubuntu20.04
      target: x86_64
      modules: []
      environment: {}
      extra_rpaths: []

  specs:
  - zlib
  - bricks ~cuda
```

**spack remove** fails to remove `bricks ~cuda`
```
$> spack -e . concretize -f
$> spack -e . remove bricks~cuda
==> Error: cannot remove bricks~cuda from /e4s-ci-command/tmp/spack.yaml, no such spec exists
```

### Error message

```
$> spack -d --stacktrace -e . remove bricks~cuda
...
lib/spack/spack/main.py:1026 ==> [2023-08-10-20:55:38.738885] SpackEnvironmentError: cannot remove bricks~cuda from /e4s-ci-command/tmp/spack.yaml, no such spec exists
lib/spack/spack/error.py:52 ==> [2023-08-10-20:55:38.739118] Error: cannot remove bricks~cuda from /e4s-ci-command/tmp/spack.yaml, no such spec exists
Traceback (most recent call last):
  File "/spack/lib/spack/spack/environment/environment.py", line 2704, in remove_user_spec
    self.pristine_configuration["specs"].remove(user_spec)
  File "/usr/lib/python3.8/_collections_abc.py", line 997, in remove
    del self[self.index(value)]
  File "/usr/lib/python3.8/_collections_abc.py", line 911, in index
    raise ValueError
ValueError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/spack/lib/spack/spack/main.py", line 1023, in main
    return _main(argv)
  File "/spack/lib/spack/spack/main.py", line 978, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/spack/lib/spack/spack/main.py", line 1006, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/spack/lib/spack/spack/main.py", line 647, in _invoke_command
    return_val = command(parser, args)
  File "/spack/lib/spack/spack/cmd/remove.py", line 41, in remove
    env.remove(spec, args.list_name, force=args.force)
  File "/spack/lib/spack/spack/environment/environment.py", line 1217, in remove
    self.manifest.remove_user_spec(str(spec))
  File "/spack/lib/spack/spack/environment/environment.py", line 2710, in remove_user_spec
    raise SpackEnvironmentError(msg) from e
spack.environment.environment.SpackEnvironmentError: cannot remove bricks~cuda from /e4s-ci-command/tmp/spack.yaml, no such spec exists
```

@becker33 

### Information on your system

* **Spack:** 0.21.0.dev0 (9438cac219cc7ff127e03bc8a09dcb5d67124bb7)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-cascadelake
* **Concretizer:** clingo

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Confirmed as of:

* **Spack:** 0.21.0.dev0 (ee6ae402aa46b6742e9f4f2041e232caa3af5a8f)
* **Python:** 3.8.10
* **Platform:** linux-ubuntu20.04-icelake
* **Concretizer:** clingo

As a note, this is due to some normalization of the spec string and removal based on string match. It won't happen when using `spack add` instead of manual editing.
makes sense to just use satisfies, so you can write `spack remove bricks`
Not sure about that:
```yaml
spack:
  specs:
  - bricks+foo
  - bricks
  - bricks~bar
```
Right now with exact match semantic we can remove the middle one without touching the other two.

Resolving patch:
diff --git a/lib/spack/spack/environment/environment.py b/lib/spack/spack/environment/environment.py
--- a/lib/spack/spack/environment/environment.py
+++ b/lib/spack/spack/environment/environment.py
@@ -2664,6 +2664,26 @@ def __init__(self, manifest_dir: Union[pathlib.Path, str]) -> None:
         self.yaml_content = with_defaults_added
         self.changed = False
 
+    def _all_matches(self, user_spec: str) -> List[str]:
+        """Maps the input string to the first equivalent user spec in the manifest,
+        and returns it.
+
+        Args:
+            user_spec: user spec to be found
+
+        Raises:
+            ValueError: if no equivalent match is found
+        """
+        result = []
+        for yaml_spec_str in self.pristine_configuration["specs"]:
+            if Spec(yaml_spec_str) == Spec(user_spec):
+                result.append(yaml_spec_str)
+
+        if not result:
+            raise ValueError(f"cannot find a spec equivalent to {user_spec}")
+
+        return result
+
     def add_user_spec(self, user_spec: str) -> None:
         """Appends the user spec passed as input to the list of root specs.
 
@@ -2684,8 +2704,9 @@ def remove_user_spec(self, user_spec: str) -> None:
             SpackEnvironmentError: when the user spec is not in the list
         """
         try:
-            self.pristine_configuration["specs"].remove(user_spec)
-            self.configuration["specs"].remove(user_spec)
+            for key in self._all_matches(user_spec):
+                self.pristine_configuration["specs"].remove(key)
+                self.configuration["specs"].remove(key)
         except ValueError as e:
             msg = f"cannot remove {user_spec} from {self}, no such spec exists"
             raise SpackEnvironmentError(msg) from e
diff --git a/lib/spack/spack/spec_list.py b/lib/spack/spack/spec_list.py
--- a/lib/spack/spack/spec_list.py
+++ b/lib/spack/spack/spec_list.py
@@ -97,8 +97,10 @@ def remove(self, spec):
             msg += "Either %s is not in %s or %s is " % (spec, self.name, spec)
             msg += "expanded from a matrix and cannot be removed directly."
             raise SpecListError(msg)
-        assert len(remove) == 1
-        self.yaml_list.remove(remove[0])
+
+        # Remove may contain more than one string representation of the same spec
+        for item in remove:
+            self.yaml_list.remove(item)
 
         # invalidate cache variables when we change the list
         self._expanded_list = None


Test patch:
diff --git a/lib/spack/spack/test/env.py b/lib/spack/spack/test/env.py
--- a/lib/spack/spack/test/env.py
+++ b/lib/spack/spack/test/env.py
@@ -13,7 +13,11 @@
 
 import spack.environment as ev
 import spack.spec
-from spack.environment.environment import SpackEnvironmentViewError, _error_on_nonempty_view_dir
+from spack.environment.environment import (
+    EnvironmentManifestFile,
+    SpackEnvironmentViewError,
+    _error_on_nonempty_view_dir,
+)
 
 pytestmark = pytest.mark.not_on_windows("Envs are not supported on windows")
 
@@ -623,3 +627,66 @@ def test_requires_on_virtual_and_potential_providers(
         assert mpileaks.satisfies("^mpich2")
         assert mpileaks["mpi"].satisfies("mpich2")
         assert not mpileaks.satisfies(f"^{possible_mpi_spec}")
+
+
+@pytest.mark.regression("39387")
+@pytest.mark.parametrize(
+    "spec_str", ["mpileaks +opt", "mpileaks  +opt   ~shared", "mpileaks  ~shared   +opt"]
+)
+def test_manifest_file_removal_works_if_spec_is_not_normalized(tmp_path, spec_str):
+    """Tests that we can remove a spec from a manifest file even if its string
+    representation is not normalized.
+    """
+    manifest = tmp_path / "spack.yaml"
+    manifest.write_text(
+        f"""\
+spack:
+  specs:
+  - {spec_str}
+"""
+    )
+    s = spack.spec.Spec(spec_str)
+    spack_yaml = EnvironmentManifestFile(tmp_path)
+    # Doing a round trip str -> Spec -> str normalizes the representation
+    spack_yaml.remove_user_spec(str(s))
+    spack_yaml.flush()
+
+    assert spec_str not in manifest.read_text()
+
+
+@pytest.mark.regression("39387")
+@pytest.mark.parametrize(
+    "duplicate_specs,expected_number",
+    [
+        # Swap variants, versions, etc. add spaces
+        (["foo +bar ~baz", "foo ~baz    +bar"], 3),
+        (["foo @1.0 ~baz %gcc", "foo ~baz @1.0%gcc"], 3),
+        # Item 1 and 3 are exactly the same
+        (["zlib +shared", "zlib      +shared", "zlib +shared"], 4),
+    ],
+)
+def test_removing_spec_from_manifest_with_exact_duplicates(
+    duplicate_specs, expected_number, tmp_path
+):
+    """Tests that we can remove exact duplicates from a manifest file.
+
+    Note that we can't get in a state with duplicates using only CLI, but this might happen
+    on user edited spack.yaml files.
+    """
+    manifest = tmp_path / "spack.yaml"
+    manifest.write_text(
+        f"""\
+    spack:
+      specs: [{", ".join(duplicate_specs)} , "zlib"]
+    """
+    )
+
+    with ev.Environment(tmp_path) as env:
+        assert len(env.user_specs) == expected_number
+        env.remove(duplicate_specs[0])
+        env.write()
+
+    assert "+shared" not in manifest.read_text()
+    assert "zlib" in manifest.read_text()
+    with ev.Environment(tmp_path) as env:
+        assert len(env.user_specs) == 1