Repository: spack/spack

Issue:
Allow extra_instructions at beginning of container stages
### Summary

Some sites may need to add SSL certificates or other things to the beginning of the stages generated by "spack containerize". There are currently "extra_instructions:build" and "extra_instructions:final" options available to add extra commands at the end of each build stage. This request is asking for similar functionality for the beginning of each build stage.

For example:

```
FROM spack/centos7:latest as builder

# Beginning extra_instructions
ADD site_specific_cert.crt /etc/pki/ca-trust/source/anchors/site_specific_cert.crt
RUN update-ca-trust

# Auto-generated content by "spack containerize"

# Ending extra_instructions
RUN yum install -y libgomp

ENTRYPOINT ["/bin/bash", "--rcfile", "/etc/profile", "-l"]
```

### Rationale

Some sites require a site-specific SSL certificate to be installed in order to reach upstream repos outside of the site. Without these certs in place, the "docker build ." fails.

### Description

Add extra_instructions before the Dockerfile contents generated by Spack. Such a feature already exists for extra_instructions after the auto-generated content.

### Additional information

Using spack developer brach:

$ spack --version
0.16.2-4259-5d23638fdc


### General information

- [X] I have run `spack --version` and reported the version of Spack
- [X] I have searched the issues of this repo and believe this is not a duplicate
Allow extra_instructions at beginning of container build and final stages
## Description
This PR adds `pre_build` and `pre_final` extra instructions to `Dockerfile` and `singularity.def`. Credits go to @kgerheiser for suggesting a first version of these changes to me.

As described in https://github.com/spack/spack/issues/26036, it's sometimes necessary to have extra instructions before the build stage. For example, when adding new `apt` sources and installing additional packages.

Likewise, I found it necessary to have extra instructions before the final stage, for example to set environment variables such as `TZ` to avoid linux configuration tools like `debconf` waiting for user input.

## Testing

I tested this extensively with `Dockerfile` for both `pre_build` and `pre_final`, and recently also with `singularity.def`.

Note. I have not updated the documentation yet. I would like to get feedback from the developers first before I go and update the documentation.

## Issues

Fixes https://github.com/spack/spack/issues/26036




Discussion:
As a work-around, `spack containerize` allows you to specify a user-defined image for both the `build` and `final` step.

You can provide images with site-specific credentials for one or both of those steps.
@mark-a-potts FYI
@ktpedre FYI
Just a heads up. This is still on my radar, but I didn't have a chance to review it yet.
> Just a heads up. This is still on my radar, but I didn't have a chance to review it yet.

@alalazo A gentle reminder to take a look at this PR when you get the chance ... thanks very much in advance.
> > Just a heads up. This is still on my radar, but I didn't have a chance to review it yet.
> 
> @alalazo A gentle reminder to take a look at this PR when you get the chance ... thanks very much in advance.

@alalazo Sorry to ping you again about this PR ...
Let's get to the bottom of this (and sorry for the slow review!). I checked the implementation and it LGTM. At a high level though I wonder if we could take another approach, i.e. expose the template being used to render the `Dockerfile` or the `singularity.def` file and give that template some extendable blocks.

In that way we can deprecate the current `final` and `build` extra instructions and leave in the YAML file only a reference to the template being used. The template itself will instead extend the blocks similarly to what is done [here](https://jinja.palletsprojects.com/en/3.1.x/templates/#child-template). Would an approach like that fit your use cases?

EDIT: forgot to mention that the appealing part of this other approach, to me, is that we'll remove extraneous verbatim line to the `spack.yaml` file, and we'll move them to where they belong i.e. the Dockerfile.
> Let's get to the bottom of this (and sorry for the slow review!). I checked the implementation and it LGTM. At a high level though I wonder if we could take another approach, i.e. expose the template being used to render the `Dockerfile` or the `singularity.def` file and give that template some extendable blocks.
> 
> In that way we can deprecate the current `final` and `build` extra instructions and leave in the YAML file only a reference to the template being used. The template itself will instead extend the blocks similarly to what is done [here](https://jinja.palletsprojects.com/en/3.1.x/templates/#child-template). Would an approach like that fit your use cases?
> 
> EDIT: forgot to mention that the appealing part of this other approach, to me, is that we'll remove extraneous verbatim line to the `spack.yaml` file, and we'll move them to where they belong i.e. the Dockerfile.

Yes, that sounds like a very good idea, thanks. This is more flexible and will not only suit our needs, but also that of others.
@climbfuji Do you want to take a stab at it? If so, I'll be happy to assist if you need any help.
> @climbfuji Do you want to take a stab at it? If so, I'll be happy to assist if you need any help.

It will take me a bit to get to this, but I can try. I'll make the changes on top of this branch so that the updates merge back cleanly into our fork (where we are using the changes in this PR already).
Ok, sounds good to me. Feel free to ping me at any time. Now that we got past release I should be a bit more responsive :sweat_smile: 
> Ok, sounds good to me. Feel free to ping me at any time. Now that we got past release I should be a bit more responsive 😅

past release is before the next release ;-) our next release is coming up in less than a month ...
@alalazo I am having a hard time finding time to work on this. Wondering if we could merge those changes as they are (you said they look good to you) and create an issue to improve the method as you described above).

Resolving patch:
diff --git a/lib/spack/spack/container/__init__.py b/lib/spack/spack/container/__init__.py
--- a/lib/spack/spack/container/__init__.py
+++ b/lib/spack/spack/container/__init__.py
@@ -39,10 +39,10 @@ def validate(configuration_file):
     # Ensure we have a "container" attribute with sensible defaults set
     env_dict = ev.config_dict(config)
     env_dict.setdefault(
-        "container", {"format": "docker", "images": {"os": "ubuntu:18.04", "spack": "develop"}}
+        "container", {"format": "docker", "images": {"os": "ubuntu:22.04", "spack": "develop"}}
     )
     env_dict["container"].setdefault("format", "docker")
-    env_dict["container"].setdefault("images", {"os": "ubuntu:18.04", "spack": "develop"})
+    env_dict["container"].setdefault("images", {"os": "ubuntu:22.04", "spack": "develop"})
 
     # Remove attributes that are not needed / allowed in the
     # container recipe
diff --git a/lib/spack/spack/container/writers/__init__.py b/lib/spack/spack/container/writers/__init__.py
--- a/lib/spack/spack/container/writers/__init__.py
+++ b/lib/spack/spack/container/writers/__init__.py
@@ -7,6 +7,7 @@
 """
 import collections
 import copy
+from typing import Optional
 
 import spack.environment as ev
 import spack.schema.env
@@ -131,6 +132,9 @@ class PathContext(tengine.Context):
     directly via PATH.
     """
 
+    # Must be set by derived classes
+    template_name: Optional[str] = None
+
     def __init__(self, config, last_phase):
         self.config = ev.config_dict(config)
         self.container_config = self.config["container"]
@@ -146,6 +150,10 @@ def __init__(self, config, last_phase):
         # Record the last phase
         self.last_phase = last_phase
 
+    @tengine.context_property
+    def depfile(self):
+        return self.container_config.get("depfile", False)
+
     @tengine.context_property
     def run(self):
         """Information related to the run image."""
@@ -280,7 +288,8 @@ def render_phase(self):
     def __call__(self):
         """Returns the recipe as a string"""
         env = tengine.make_environment()
-        t = env.get_template(self.template_name)
+        template_name = self.container_config.get("template", self.template_name)
+        t = env.get_template(template_name)
         return t.render(**self.to_dict())
 
 
diff --git a/lib/spack/spack/schema/container.py b/lib/spack/spack/schema/container.py
--- a/lib/spack/spack/schema/container.py
+++ b/lib/spack/spack/schema/container.py
@@ -63,6 +63,8 @@
         },
         # Add labels to the image
         "labels": {"type": "object"},
+        # Use a custom template to render the recipe
+        "template": {"type": "string", "default": None},
         # Add a custom extra section at the bottom of a stage
         "extra_instructions": {
             "type": "object",
@@ -82,6 +84,16 @@
             },
         },
         "docker": {"type": "object", "additionalProperties": False, "default": {}},
+        "depfile": {"type": "boolean", "default": False},
+    },
+    "deprecatedProperties": {
+        "properties": ["extra_instructions"],
+        "message": (
+            "container:extra_instructions has been deprecated and will be removed "
+            "in Spack v0.21. Set container:template appropriately to use custom Jinja2 "
+            "templates instead."
+        ),
+        "error": False,
     },
 }
 


Test patch: