Repository: spack/spack

Issue:
containerize: python+pythoncmd fails to produce working symlink for pythoncmd
When installing python+pythoncmd as part of an environment and exporting it to a container, the python symlink in the container is broken.  A minimal reproducer is listed below. 

For a given spack environment file: 
```
sajid@LAPTOP-CDJT2P3R ~/t/container> cat spack.yaml
spack:
  specs:
  - python+pythoncmd
  view: true
  container:
    format: docker
    images:
      os: ubuntu:22.04
      spack: develop
    os_packages:
      build:
      - build-essential
      - bash
      final:
      - build-essential
      - bash
```

The spack generated Dockerfile is: 
```
sajid@LAPTOP-CDJT2P3R ~/t/container> cat Dockerfile
# Build stage with Spack pre-installed and ready to be used
FROM spack/ubuntu-jammy:latest as builder

# Install OS packages needed to build the software
RUN apt-get -yqq update && apt-get -yqq upgrade \
 && apt-get -yqq install build-essential bash \
 && rm -rf /var/lib/apt/lists/*

# What we want to install and how we want to install it
# is specified in a manifest file (spack.yaml)
RUN mkdir /opt/spack-environment \
&&  (echo "spack:" \
&&   echo "  specs:" \
&&   echo "  - python+pythoncmd" \
&&   echo "  view: /opt/view" \
&&   echo "  concretization: together" \
&&   echo "  config:" \
&&   echo "    install_tree: /opt/software") > /opt/spack-environment/spack.yaml

# Install the software, remove unnecessary deps
RUN cd /opt/spack-environment && \
    spack env activate . && \
    spack install --fail-fast && \
    spack gc -y

# Strip all the binaries
RUN find -L /opt/view/* -type f -exec readlink -f '{}' \; | \
    xargs file -i | \
    grep 'charset=binary' | \
    grep 'x-executable\|x-archive\|x-sharedlib' | \
    awk -F: '{print $1}' | xargs strip -s

# Modifications to the environment that are necessary to run
RUN cd /opt/spack-environment && \
    spack env activate --sh -d . >> /etc/profile.d/z10_spack_environment.sh

# Bare OS image to run the installed executables
FROM ubuntu:22.04

COPY --from=builder /opt/spack-environment /opt/spack-environment
COPY --from=builder /opt/software /opt/software
COPY --from=builder /opt/view /opt/view
COPY --from=builder /etc/profile.d/z10_spack_environment.sh /etc/profile.d/z10_spack_environment.sh

RUN apt-get -yqq update && apt-get -yqq upgrade \
 && apt-get -yqq install build-essential bash \
 && rm -rf /var/lib/apt/lists/*
ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```

And the generated container does not have a working python command as demonstrated below: 
```
sajid@LAPTOP-CDJT2P3R ~/t/container> docker run --rm -it (docker build -q .)
root@38a69f974a42:/# . /etc/profile.d/z10_spack_environment.sh
root@38a69f974a42:/# which python
root@38a69f974a42:/# which python3.9
/opt/view/bin/python3.9
root@38a69f974a42:/# ls -al /opt/view/bin/ | grep python
lrwxrwxrwx  1 root root    58 Jun  2 17:01 python -> /opt/._view/fjfihhazh6sdd6uv3s6s4r2wuptk7tjd/bin/python3.9
lrwxrwxrwx  1 root root    65 Jun  2 17:01 python-config -> /opt/._view/fjfihhazh6sdd6uv3s6s4r2wuptk7tjd/bin/python3.9-config
lrwxrwxrwx  1 root root    58 Jun  2 17:01 python3 -> /opt/._view/fjfihhazh6sdd6uv3s6s4r2wuptk7tjd/bin/python3.9
lrwxrwxrwx  1 root root    65 Jun  2 17:01 python3-config -> /opt/._view/fjfihhazh6sdd6uv3s6s4r2wuptk7tjd/bin/python3.9-config
-rwxr-xr-x  1 root root 17688 Jun  2 17:01 python3.9
-rwxr-xr-x  1 root root  3098 Jun  2 17:01 python3.9-config
-rwxr-xr-x  1 root root 65496 Jun  2 17:01 python3.9-gdb.py
root@38a69f974a42:/#
```

### General information

- [X] I have run `spack debug report` and reported the version of Spack/Python/Platform
- [X] I have searched the issues of this repo and believe this is not a duplicate
- [X] I have run the failing commands in debug mode and reported the output

```
sajid@LAPTOP-CDJT2P3R ~/t/container> spack debug report
* **Spack:** 0.19.0.dev0 (f9dfd5fcb88a22c2a686409fa15b3724de52c4cc)
* **Python:** 3.10.4
* **Platform:** linux-ubuntu22.04-zen2
* **Concretizer:** clingo
sajid@LAPTOP-CDJT2P3R ~/t/container>
```


Discussion:
This was originally observed when building a container using GithubActions:
- The workflow to build the container is [here](https://github.com/fnalacceleratormodeling/synergia2-containers/blob/main/.github/workflows/docker-publish.yml).
- The spack environment is [here](https://github.com/fnalacceleratormodeling/synergia2-containers/blob/main/ubuntu-gcc/spack.yaml)
- A lightly edited version of the spack generated `Dockerfile` used in the workflow is [here](https://github.com/fnalacceleratormodeling/synergia2-containers/blob/main/ubuntu-gcc/Dockerfile)

The python installation copied into the container has a working symlink:
```
root@3c6ac8aaaadd:/opt/view/bin# ls -al /opt/software/linux-ubuntu22.04-x86_64_v2/gcc-11.2.0/python-3.9.12-o7jxqzh2qm5mhjjtiz4nwnaowlrzx3nh/bin/
total 124
drwxr-sr-x 2 root root  4096 Jun  2 15:27 .
drwxr-sr-x 7 root root  4096 Jun  2 15:26 ..
lrwxrwxrwx 1 root root     8 Jun  2 15:26 2to3 -> 2to3-3.9
-rwxr-xr-x 1 root root   190 Jun  2 15:26 2to3-3.9
lrwxrwxrwx 1 root root     7 Jun  2 15:26 idle3 -> idle3.9
-rwxr-xr-x 1 root root   188 Jun  2 15:26 idle3.9
-rwxr-xr-x 1 root root   318 Jun  2 15:27 pip3
-rwxr-xr-x 1 root root   318 Jun  2 15:27 pip3.9
lrwxrwxrwx 1 root root     8 Jun  2 15:26 pydoc3 -> pydoc3.9
-rwxr-xr-x 1 root root   173 Jun  2 15:26 pydoc3.9
lrwxrwxrwx 1 root root   111 Jun  2 15:27 python -> /opt/software/linux-ubuntu22.04-x86_64_v2/gcc-11.2.0/python-3.9.12-o7jxqzh2qm5mhjjtiz4nwnaowlrzx3nh/bin/python3
lrwxrwxrwx 1 root root   118 Jun  2 15:27 python-config -> /opt/software/linux-ubuntu22.04-x86_64_v2/gcc-11.2.0/python-3.9.12-o7jxqzh2qm5mhjjtiz4nwnaowlrzx3nh/bin/python3-config
lrwxrwxrwx 1 root root     9 Jun  2 15:26 python3 -> python3.9
lrwxrwxrwx 1 root root    16 Jun  2 15:26 python3-config -> python3.9-config
-rwxr-xr-x 1 root root 17688 Jun  2 15:26 python3.9
-rwxr-xr-x 1 root root  3153 Jun  2 15:26 python3.9-config
-rwxr-xr-x 1 root root 65496 Jun  2 15:27 python3.9-gdb.py
root@3c6ac8aaaadd:/opt/view/bin# /opt/software/linux-ubuntu22.04-x86_64_v2/gcc-11.2.0/python-3.9.12-o7jxqzh2qm5mhjjtiz4nwnaowlrzx3nh/bin/python
Python 3.9.12 (main, Jun  2 2022, 15:26:11)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
root@3c6ac8aaaadd:/opt/view/bin#
```

While a python symlink exists in the view, it is not functional (but python3.9 is!):
```
root@3c6ac8aaaadd:/# ls -al /opt/view/bin/ | grep python
lrwxrwxrwx  1 root root    58 Jun  2 16:24 python -> /opt/._view/rc463wzcbo623bm2u4mbbudkl22latmt/bin/python3.9
lrwxrwxrwx  1 root root    65 Jun  2 16:24 python-config -> /opt/._view/rc463wzcbo623bm2u4mbbudkl22latmt/bin/python3.9-config
lrwxrwxrwx  1 root root    58 Jun  2 16:24 python3 -> /opt/._view/rc463wzcbo623bm2u4mbbudkl22latmt/bin/python3.9
lrwxrwxrwx  1 root root    65 Jun  2 16:24 python3-config -> /opt/._view/rc463wzcbo623bm2u4mbbudkl22latmt/bin/python3.9-config
-rwxr-xr-x  1 root root 17688 Jun  2 16:24 python3.9
-rwxr-xr-x  1 root root  3098 Jun  2 16:24 python3.9-config
-rwxr-xr-x  1 root root 65496 Jun  2 16:24 python3.9-gdb.py
root@3c6ac8aaaadd:/# /opt/view/bin/python
bash: /opt/view/bin/python: No such file or directory
root@3c6ac8aaaadd:/# /opt/view/bin/python3.9
Python 3.9.12 (main, Jun  2 2022, 15:26:11)
[GCC 11.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>
root@3c6ac8aaaadd:/#
```

### Information on your system

Using spack from `spack/ubuntu-jammy:latest` as of today via GithubActions. 



I am also hitting this issue.  I can add that py-* packages that create an executable shim file, like, say, py-cutadapt, also have incorrect shebang lines:

```
# head -n 1 /opt/view/bin/cutadapt 
#!/opt/._view/3ddsdr3hhivd3xyxydzurx5wyd5qrgvk/bin/python3.7
```

which causes:
```
/opt/view/bin/cutadapt: /opt/._view/3ddsdr3hhivd3xyxydzurx5wyd5qrgvk/bin/python3.7: bad interpreter: No such file or directory
```
So, here's what's going on....

1. Views aren't built in the directory named by the view, they're built in directories with obfuscated names that are symlinked to the actual name of the view directory.  This results in an even more confusing hidden name because containerize uses a view named `view`.  E.g.

   ``` shellsession
   [root@e9e5c68c7f38 ~]# ls -la /opt
   total 0
   drwxr-xr-x. 1 root root 35 Jun 24 17:53 .
   drwxr-xr-x. 1 root root  6 Jun 24 20:45 ..
   drwxr-xr-x. 1 root root 46 Jun 24 17:53 ._view
   drwxr-xr-x. 1 root root 23 Jun 24 16:56 software
   drwxr-xr-x. 1 root root 17 Jun 23 01:57 spack
   drwxr-xr-x. 1 root root 24 Jun 24 17:53 spack-environment
   lrwxrwxrwx. 1 root root 44 Jun 24 17:53 view -> /opt/._view/3ddsdr3hhivd3xyxydzurx5wyd5qrgvk
   ```

   [Code here, with illuminating comment](https://github.com/spack/spack/blob/develop/lib/spack/spack/environment/environment.py#L530-L540)

2. The Dockerfile that containerize creates copies `/opt/view`, not the hidden dir:

   ``` Dockerfile
   [...]
   COPY --from=builder /opt/view /opt/view
   [...]
   ```

3. The bit that adds the shebang line apparently resolves the symlink.

As an interim fix, something like this seems to work:

1. Add an bit of code to your `spack.yaml` that recreates the link

``` Dockerfile
    extra_instructions:
      build: |
        # other things might happen here
        RUN echo /opt/._view/* > /tmp/funky-view-link-name
      final: |
        # other things might happen here
        RUN mkdir /opt/._view \
          && ln -s /opt/view $(cat /tmp/funky-view-link-name)
```

A real fix might be for containerize *to copy the hidden dir and then recreate the `/opt/view` symlink*.  It [seems like the copy command supports `*`][wildcard], so

```
# untested...
RUN mkdir /opt/._view
COPY --from=builder /opt/._view/* /opt/._view_
RUN ln -s /opt/view /opt/._view/*
```

might work.

@alalazo and/or all y'all, I'd love {cr,w}iticism.

Sadly, I'm still not allowed to push code....

[wildcard]: https://stackoverflow.com/a/49940132
Thanks for triaging that @hartzell ! What you say makes perfect sense. The addition of the indirection for views has been added after `spack containerize` and I guess modifying the template for containers was overlooked :cry: 

Resolving patch:
diff --git a/lib/spack/spack/container/writers/__init__.py b/lib/spack/spack/container/writers/__init__.py
--- a/lib/spack/spack/container/writers/__init__.py
+++ b/lib/spack/spack/container/writers/__init__.py
@@ -171,11 +171,12 @@ def strip(self):
     def paths(self):
         """Important paths in the image"""
         Paths = collections.namedtuple('Paths', [
-            'environment', 'store', 'view'
+            'environment', 'store', 'hidden_view', 'view'
         ])
         return Paths(
             environment='/opt/spack-environment',
             store='/opt/software',
+            hidden_view='/opt/._view',
             view='/opt/view'
         )
 


Test patch: