Repository: spack/spack

Issue:
`spack develop` trips up over dead links
### Steps to reproduce

Reproduce with:
```console
spack env create zlib
spack env activate zlib
spack add zlib@1.2.12
dir=$(mktemp -d)
rm -r ${dir}
spack develop -p ${dir} zlib@1.2.12
spack install
ln -sf ${dir}/file-does-not-exist ${dir}/link-to-missing-file
spack install
```

Backout with:
```console
spack env deactivate
spack env rm -y zlib
rm -r ${dir}
```

### Error message

```console
==> Created environment 'zlib' in /home/wdconinc/git/spack/var/spack/environments/zlib
==> You can activate this environment with:
==>   spack env activate zlib
==> Adding zlib@1.2.12 to environment zlib
==> Configuring spec zlib@1.2.12 for development at path /tmp/tmp.4d0Ns3M7wo
==> Fetching https://mirror.spack.io/_source-cache/archive/91/91844808532e5ce316b3c010929493c0244f3d37593afd6de04f71821d5136d9.tar.gz
==> Starting concretization
==> Environment concretized in 0.95 seconds.
==> Concretized zlib@1.2.12
 -   utp562b  zlib@1.2.12%gcc@12.1.0+optimize+pic+shared dev_path=/tmp/tmp.4d0Ns3M7wo patches=0d38234 arch=linux-ubuntu22.04-skylake

==> Installing environment zlib
==> Installing zlib-1.2.12-utp562bvdu27t5b2ewscvkng3cpxlmey
==> No binary for zlib-1.2.12-utp562bvdu27t5b2ewscvkng3cpxlmey found: installing from source
==> Applied patch /home/wdconinc/git/spack/var/spack/repos/builtin/packages/zlib/configure-cc.patch
==> zlib: Executing phase: 'install'
==> zlib: Successfully installed zlib-1.2.12-utp562bvdu27t5b2ewscvkng3cpxlmey
  Fetch: 0.00s.  Build: 0.82s.  Total: 0.82s.
[+] /opt/software/linux-ubuntu22.04-skylake/gcc-12.1.0/zlib-1.2.12-utp562bvdu27t5b2ewscvkng3cpxlmey
==> Updating view at /home/wdconinc/git/spack/var/spack/environments/zlib/.spack-env/view
==> Installing environment zlib
==> Error: [Errno 2] No such file or directory: '/tmp/tmp.4d0Ns3M7wo/link-to-missing-file'
```

Debug output (abridged):
```console
==> [2022-08-15-18:38:47.916728] Installing environment zlib
==> [2022-08-15-18:38:47.916778] Assessing installation status of environment packages
==> [2022-08-15-18:38:48.021639] Processing 1 uninstalled specs
Traceback (most recent call last):
  File "/home/wdconinc/git/spack/bin/spack", line 100, in <module>
    sys.exit(spack.main.main())
  File "/home/wdconinc/git/spack/lib/spack/spack/main.py", line 961, in main
    return _main(argv)
  File "/home/wdconinc/git/spack/lib/spack/spack/main.py", line 916, in _main
    return finish_parse_and_run(parser, cmd_name, env_format_error)
  File "/home/wdconinc/git/spack/lib/spack/spack/main.py", line 944, in finish_parse_and_run
    return _invoke_command(command, parser, args, unknown)
  File "/home/wdconinc/git/spack/lib/spack/spack/main.py", line 608, in _invoke_command
    return_val = command(parser, args)
  File "/home/wdconinc/git/spack/lib/spack/spack/cmd/install.py", line 483, in install
    install_all_specs_from_active_environment(
  File "/home/wdconinc/git/spack/lib/spack/spack/cmd/install.py", line 385, in install_all_specs_from_active_environment
    env.install_all(**install_kwargs)
  File "/home/wdconinc/git/spack/lib/spack/spack/environment/environment.py", line 1602, in install_all
    self.install_specs(None, **install_args)
  File "/home/wdconinc/git/spack/lib/spack/spack/environment/environment.py", line 1631, in install_specs
    specs_to_overwrite = self._get_overwrite_specs()
  File "/home/wdconinc/git/spack/lib/spack/spack/environment/environment.py", line 1550, in _get_overwrite_specs
    changed_dev_specs = set(s for s in specs_to_check if _is_dev_spec_and_has_changed(s))
  File "/home/wdconinc/git/spack/lib/spack/spack/environment/environment.py", line 1550, in <genexpr>
    changed_dev_specs = set(s for s in specs_to_check if _is_dev_spec_and_has_changed(s))
  File "/home/wdconinc/git/spack/lib/spack/spack/environment/environment.py", line 321, in _is_dev_spec_and_has_changed
    mtime = fs.last_modification_time_recursive(dev_path_var.value)
  File "/home/wdconinc/git/spack/lib/spack/spack/util/path.py", line 151, in path_filter_caller
    return func(*args, **kwargs)
  File "/home/wdconinc/git/spack/lib/spack/llnl/util/filesystem.py", line 1295, in last_modification_time_recursive
    times.extend(
  File "/home/wdconinc/git/spack/lib/spack/llnl/util/filesystem.py", line 1296, in <genexpr>
    os.stat(os.path.join(root, name)).st_mtime
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/tmp.4d0Ns3M7wo/link-to-missing-file'
```

### Information on your system

* **Spack:** 0.19.0.dev0 (cd80779738ea5e31cc27dd285b75f8ddea8c096f)
* **Python:** 3.10.6
* **Platform:** linux-ubuntu22.04-skylake
* **Concretizer:** clingo


### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output


Discussion:
Addressed by
```diff
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
index a690ef3c37..d9b7d47e4b 100644
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -1296,6 +1296,7 @@ def last_modification_time_recursive(path):
         os.stat(os.path.join(root, name)).st_mtime
         for root, dirs, files in os.walk(path)
         for name in dirs + files
+        if os.path.isfile(os.path.join(root, name)) or os.path.isdir(os.path.join(root, name))
     )
     return max(times)
 
```
but that generator expression is starting to suffer readability.

Resolving patch:
diff --git a/lib/spack/llnl/util/filesystem.py b/lib/spack/llnl/util/filesystem.py
--- a/lib/spack/llnl/util/filesystem.py
+++ b/lib/spack/llnl/util/filesystem.py
@@ -1293,7 +1293,7 @@ def last_modification_time_recursive(path):
     path = os.path.abspath(path)
     times = [os.stat(path).st_mtime]
     times.extend(
-        os.stat(os.path.join(root, name)).st_mtime
+        os.lstat(os.path.join(root, name)).st_mtime
         for root, dirs, files in os.walk(path)
         for name in dirs + files
     )


Test patch: