Repository: coder/envbuilder
Pull request #401 (2024-10-31 09:42:16)

feat(docs): document build secrets
This PR provides documentation for envbuilder build secrets.
Documentation for how to use build secrets from Terraform in conjunction with Coder will be provided separately.

Note that the documentation refers to `ghcr.io/coder/envbuilder:latest`. That will only work once we release this and push build secret support to ghcr. For now, you can test this by building your own version of the envbuilder container and using that instead.


Patch:
diff --git a/docs/build-secrets.md b/docs/build-secrets.md
new file mode 100644
index 0000000..00c38c7
--- /dev/null
+++ b/docs/build-secrets.md
@@ -0,0 +1,165 @@
+# Build Secrets
+
+Envbuilder supports [build secrets](https://docs.docker.com/reference/dockerfile/#run---mounttypesecret). Build secrets are useful when you need to use sensitive information during the image build process and:
+* the secrets should not be present in the built image.
+* the secrets should not be accessible in the container after its build has concluded.
+
+If your Dockerfile contains directives of the form `RUN --mount=type=secret,...`, Envbuilder will attempt to mount build secrets as specified in the directive. Unlike the `docker build` command, Envbuilder does not support the `--secret` flag. Instead, Envbuilder collects build secrets from the `ENVBUILDER_BUILD_SECRETS` environment variable. These build secrets will not be present in any cached layers or images that are pushed to an image repository. Nor will they be available at run time.
+
+## Example
+
+To illustrate build secrets in Envbuilder, let's build, push and run a container locally. These concepts will transfer to Kubernetes or other containerised environments. Note that this example is for illustrative purposes only and is not fit for production use. Production considerations are discussed in the next section.
+
+First, start a local docker registry, so that we can push and inspect the built image:
+```bash
+docker run --rm -d -p 5000:5000 --name envbuilder-registry registry:2
+```
+
+Then, prepare the files to build our container.
+```bash
+mkdir test-build-secrets
+cd test-build-secrets
+cat << EOF > Dockerfile
+FROM alpine:latest
+
+RUN --mount=type=secret,id=TEST_BUILD_SECRET_A,env=TEST_BUILD_SECRET_A echo -n \$TEST_BUILD_SECRET_A | sha256sum > /foo_secret_hash.txt
+RUN --mount=type=secret,id=TEST_BUILD_SECRET_B,dst=/tmp/bar.secret cat /tmp/bar.secret | sha256sum > /bar_secret_hash.txt
+EOF
+cat << EOF > devcontainer.json
+{
+    "build": { 
+        "dockerfile": "Dockerfile"
+    }
+}
+EOF
+echo 'runtime-secret-a' > runtime-secret.txt
+```
+
+The Dockerfile requires two build secrets: `TEST_BUILD_SECRET_A` and `TEST_BUILD_SECRET_B`. Their values are arbitrarily set to `secret-foo` and `secret-bar` by the command below. Building the container image writes the checksums for these secrets to disk. This illustrates that the secrets can be used in the build to enact side effects without exposing the secrets themselves.
+
+Execute the build using this command:
+```bash
+docker run -it --rm \
+    -e ENVBUILDER_BUILD_SECRETS='TEST_BUILD_SECRET_A=secret-foo,TEST_BUILD_SECRET_B=secret-bar' \
+    -e ENVBUILDER_INIT_SCRIPT='/bin/sh' \
+    -e ENVBUILDER_CACHE_REPO=$(docker inspect envbuilder-registry | jq -r '.[].NetworkSettings.IPAddress'):5000/test-container \
+    -e ENVBUILDER_PUSH_IMAGE=1 \
+    -v $PWD:/workspaces/empty \
+    -v $PWD/runtime-secret.txt:/runtime-secret.txt \
+    ghcr.io/coder/envbuilder:latest
+```
+
+This will result in a shell session inside the built container.
+You can now verify three things:
+
+Firstly, the secrets provided to build are not available once the container is running. They are no longer on disk, nor are they in the process environment, or in `/proc/self/environ`: 
+```bash
+cat /proc/self/environ | tr '\0' '\n'
+printenv
+```
+Expected output:
+```bash
+/workspaces/empty # cat /proc/self/environ | tr '\0' '\n'
+HOSTNAME=c0b0ee3d5564
+SHLVL=2
+HOME=/root
+TERM=xterm
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+DEVCONTAINER_CONFIG=/workspaces/empty/devcontainer.json
+ENVBUILDER=true
+TS_DEBUG_TRIM_WIREGUARD=false
+PWD=/workspaces/empty
+DEVCONTAINER=true
+/workspaces/empty # printenv
+HOSTNAME=c0b0ee3d5564
+SHLVL=2
+HOME=/root
+TERM=xterm
+PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
+DEVCONTAINER_CONFIG=/workspaces/empty/devcontainer.json
+ENVBUILDER=true
+TS_DEBUG_TRIM_WIREGUARD=false
+PWD=/workspaces/empty
+DEVCONTAINER=true
+/workspaces/empty # 
+```
+
+Secondly, the secrets were still useful during the build. The following commands show that the secrets had side effects inside the build, without remaining in the image:
+```bash
+echo -n "secret-foo" | sha256sum
+cat /foo_secret_hash.txt
+echo -n "secret-bar" | sha256sum
+cat /bar_secret_hash.txt
+```
+
+Notice that the first two checksums match and that the last two checksums match. Expected output:
+```
+/workspaces/empty # echo -n "secret-foo" | sha256sum
+9a888f08a057159d2ea8fb69d38c9a25e367d7ca3128035b7f6dee2ca988c3d8  -
+/workspaces/empty # cat /foo_secret_hash.txt
+9a888f08a057159d2ea8fb69d38c9a25e367d7ca3128035b7f6dee2ca988c3d8  -
+/workspaces/empty # echo -n "secret-bar" | sha256sum
+fb1c9d1220e429b30c60d028b882f735b5af72d7b5496d9202737fe9f1d38289  -
+/workspaces/empty # cat /bar_secret_hash.txt
+fb1c9d1220e429b30c60d028b882f735b5af72d7b5496d9202737fe9f1d38289  -
+/workspaces/empty # 
+```
+
+Thirdly, the runtime secret that was mounted as a volume is still mounted into the container and accessible. This is why volumes are inappropriate analogues to native docker build secrets. However, notice further down that this runtime secret volume's contents are not present in the built image. It is therefore safe to mount a volume into envbuilder for use during runtime without fear that it will be present in the image that envbuilder builds.
+
+Finally, exit the container:
+```bash
+exit
+```
+
+### Verifying that images are secret free
+To verify that the built image doesn't contain build secrets, run the following:
+
+```bash
+docker pull localhost:5000/test-container:latest
+docker save -o test-container.tar localhost:5000/test-container
+mkdir -p test-container
+tar -xf test-container.tar -C test-container/
+cd test-container
+# Scan image layers for secrets:
+find . -type f | xargs tar -xOf 2>/dev/null  | strings | grep -rn "secret-foo"
+find . -type f | xargs tar -xOf 2>/dev/null  | strings | grep -rn "secret-bar"
+find . -type f | xargs tar -xOf 2>/dev/null  | strings | grep -rn "runtime-secret"
+# Scan image manifests for secrets:
+find . -type f | xargs -n1 grep -rnI 'secret-foo'
+find . -type f | xargs -n1 grep -rnI 'secret-bar'
+find . -type f | xargs -n1 grep -rnI 'runtime-secret'
+cd ../
+```
+
+The output of all find/grep commands should be empty.
+To verify that it scans correctly, replace "secret-foo" with "envbuilder" and rerun the commands. It should find strings related to Envbuilder that are not secrets.
+
+### Cleanup
+
+Having verified that no secrets were included in the image, we can now delete the artifacts that we saved to disk and remove the containers.
+```bash
+cd ../
+rm -r test-build-secrets
+docker stop envbuilder-registry
+```
+
+## Security and Production Use
+The example above ignores various security concerns for the sake of simple illustration. To use build secrets securely, consider these factors:
+
+### Build Secret Purpose and Management
+Build secrets are meant for use cases where the secret should not be accessible from the built image, nor from the running container. If you need the secret at runtime, use a volume instead. Volumes that are mounted into a container will not be included in the final image, but still be available at runtime. 
+
+Build secrets are only protected if they are not copied or moved from their location as designated in the `RUN` directive. If a build secret is used, care should be taken to ensure that it is not copied or otherwise persisted into an image layer beyond the control of Envbuilder.
+
+### Who should be able to access build secrets, when and where?
+Anyone with sufficient access to attach directly to the container (eg. using `kubectl`), will be able to read build secrets if they attach to the container before it has concluded its build. Anyone with sufficient access to the platform that hosts the Envbuilder container will also be able to read these build secrets from where the platform stores them. This is true for other build systems, and containerised software in general.
+
+The secure way to use build secrets with Envbuilder is to deny users access to the platform that hosts Envbuilder. Only grant access to the Envbuilder container once it has concluded its build, using a trusted non-platform channel like ssh or the coder agent running inside the container. Once control has been handed to such a runtime container process, Envbuilder will have cleared all secrets that it set from the container.
+
+If secrets should be accessible at runtime, do not use build secrets. Rather, mount the secret data using a volume or environment variable. Envbuilder will not include mounted volumes in the image that it pushes to any cache repositories, but they will still be available to users that connect to the container.
+
+### Container Management beyond Envbuilder's control
+Container orchestration systems mount certain artifacts into containers for various reasons. It is possible that some of these might grant indirect access to build secrets. Consider kubernetes. It will mount a service account token into running containers. Depending on the access granted to this service account token, it may be possible to read build secrets and other sensitive data using the kubernetes API. This should not be possible by default, but Envbuilder cannot provide such a guarantee.
+
+When building a system that uses Envbuilder, ensure that your platform does not expose unintended secret information to the container.
\ No newline at end of file
diff --git a/docs/env-variables.md b/docs/env-variables.md
index cb7054f..6f9653b 100644
--- a/docs/env-variables.md
+++ b/docs/env-variables.md
@@ -21,6 +21,7 @@
 | `--force-safe` | `ENVBUILDER_FORCE_SAFE` |  | Ignores any filesystem safety checks. This could cause serious harm to your system! This is used in cases where bypass is needed to unblock customers. |
 | `--insecure` | `ENVBUILDER_INSECURE` |  | Bypass TLS verification when cloning and pulling from container registries. |
 | `--ignore-paths` | `ENVBUILDER_IGNORE_PATHS` |  | The comma separated list of paths to ignore when building the workspace. |
+| `--build-secrets` | `ENVBUILDER_BUILD_SECRETS` |  | The list of secret environment variables to use when building the image. |
 | `--skip-rebuild` | `ENVBUILDER_SKIP_REBUILD` |  | Skip building if the MagicFile exists. This is used to skip building when a container is restarting. e.g. docker stop -> docker start This value can always be set to true - even if the container is being started for the first time. |
 | `--git-url` | `ENVBUILDER_GIT_URL` |  | The URL of a Git repository containing a Devcontainer or Docker image to clone. This is optional. |
 | `--git-clone-depth` | `ENVBUILDER_GIT_CLONE_DEPTH` |  | The depth to use when cloning the Git repository. |
diff --git a/envbuilder.go b/envbuilder.go
index 37a97bb..324a84c 100644
--- a/envbuilder.go
+++ b/envbuilder.go
@@ -530,10 +530,6 @@ func run(ctx context.Context, opts options.Options, execArgs *execArgsInfo) erro
 				destinations = append(destinations, opts.CacheRepo)
 			}
 
-			buildSecrets := options.GetBuildSecrets(os.Environ())
-			// Ensure that build secrets do not make it into the runtime environment or the setup script:
-			options.ClearBuildSecretsFromProcessEnvironment()
-
 			kOpts := &config.KanikoOptions{
 				// Boilerplate!
 				CustomPlatform:     platforms.Format(platforms.Normalize(platforms.DefaultSpec())),
@@ -558,7 +554,7 @@ func run(ctx context.Context, opts options.Options, execArgs *execArgsInfo) erro
 				},
 				ForceUnpack:       true,
 				BuildArgs:         buildParams.BuildArgs,
-				BuildSecrets:      buildSecrets,
+				BuildSecrets:      opts.BuildSecrets,
 				CacheRepo:         opts.CacheRepo,
 				Cache:             opts.CacheRepo != "" || opts.BaseImageCacheDir != "",
 				DockerfilePath:    buildParams.DockerfilePath,
@@ -1258,6 +1254,7 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error)
 	if opts.CacheRepo != "" {
 		destinations = append(destinations, opts.CacheRepo)
 	}
+
 	kOpts := &config.KanikoOptions{
 		// Boilerplate!
 		CustomPlatform:     platforms.Format(platforms.Normalize(platforms.DefaultSpec())),
@@ -1282,6 +1279,7 @@ func RunCacheProbe(ctx context.Context, opts options.Options) (v1.Image, error)
 		},
 		ForceUnpack:       true,
 		BuildArgs:         buildParams.BuildArgs,
+		BuildSecrets:      opts.BuildSecrets,
 		CacheRepo:         opts.CacheRepo,
 		Cache:             opts.CacheRepo != "" || opts.BaseImageCacheDir != "",
 		DockerfilePath:    buildParams.DockerfilePath,
diff --git a/options/options.go b/options/options.go
index efa61be..01ed510 100644
--- a/options/options.go
+++ b/options/options.go
@@ -88,6 +88,9 @@ type Options struct {
 	// IgnorePaths is the comma separated list of paths to ignore when building
 	// the workspace.
 	IgnorePaths []string
+	// BuildSecrets is the list of secret environment variables to use when
+	// building the image.
+	BuildSecrets []string
 	// SkipRebuild skips building if the MagicFile exists. This is used to skip
 	// building when a container is restarting. e.g. docker stop -> docker start
 	// This value can always be set to true - even if the container is being
@@ -323,6 +326,12 @@ func (o *Options) CLI() serpent.OptionSet {
 			Description: "The comma separated list of paths to ignore when " +
 				"building the workspace.",
 		},
+		{
+			Flag:        "build-secrets",
+			Env:         WithEnvPrefix("BUILD_SECRETS"),
+			Value:       serpent.StringArrayOf(&o.BuildSecrets),
+			Description: "The list of secret environment variables to use " + "when building the image.",
+		},
 		{
 			Flag:  "skip-rebuild",
 			Env:   WithEnvPrefix("SKIP_REBUILD"),
diff --git a/options/secrets.go b/options/secrets.go
deleted file mode 100644
index 9896fb3..0000000
--- a/options/secrets.go
+++ /dev/null
@@ -1,46 +0,0 @@
-package options
-
-import (
-	"fmt"
-	"os"
-
-	"github.com/coder/serpent"
-)
-
-var buildSecretPrefix = fmt.Sprintf("%sBUILD_SECRET_", envPrefix)
-
-// EnvWithBuildSecretPrefix returns a string in the format of a build secret environment variable.
-func EnvWithBuildSecretPrefix(secretName, secretValue string) string {
-	return fmt.Sprintf("%s%s=%s", buildSecretPrefix, secretName, secretValue)
-}
-
-// GetBuildSecrets sources build secrets from the given environment.
-//
-// In a normal docker build, build secrets would be passed in via the
-// `docker build --secret` flag. envbuilder is more analogous to a
-// `docker run` that just happens to build its own container. It doesn't have
-// access to the `--secret` flag. As an alternative, we source these from the
-// envbuilder process environment.
-func GetBuildSecrets(environ []string) []string {
-	buildSecrets := serpent.ParseEnviron(environ, buildSecretPrefix).ToOS()
-	return buildSecrets
-}
-
-// ClearBuildSecretsFromProcessEnvironment unsets all build secrets from the process environment.
-// NOTE: This does not remove them from /proc/self/environ. They are still visible
-// there unless execve(2) is called.
-//
-// Unlike runtime secrets in the devcontainer spec or orchestration systems like
-// Kubernetes, build secrets should not be available at run time. envbuilder blurs
-// the line between build time and run time by transitioning from one to the other
-// within the same process in the same container.
-//
-// These build secrets should not make it into the runtime environment of the runtime
-// container init process. It is therefore useful to unset build secret environment
-// variables to ensure they aren't accidentally passed into the exec call.
-func ClearBuildSecretsFromProcessEnvironment() {
-	buildSecrets := serpent.ParseEnviron(os.Environ(), buildSecretPrefix)
-	for _, secret := range buildSecrets {
-		os.Unsetenv(buildSecretPrefix + secret.Name)
-	}
-}


Test patch:
diff --git a/integration/integration_test.go b/integration/integration_test.go
index 2c12b97..204ff90 100644
--- a/integration/integration_test.go
+++ b/integration/integration_test.go
@@ -1116,36 +1116,6 @@ func TestUnsetOptionsEnv(t *testing.T) {
 	}
 }
 
-func TestUnsetSecretEnvs(t *testing.T) {
-	t.Parallel()
-
-	// Ensures that a Git repository with a devcontainer.json is cloned and built.
-	srv := gittest.CreateGitServer(t, gittest.Options{
-		Files: map[string]string{
-			".devcontainer/devcontainer.json": `{
-				"name": "Test",
-				"build": {
-					"dockerfile": "Dockerfile"
-				},
-			}`,
-			".devcontainer/Dockerfile": "FROM " + testImageAlpine + "\nENV FROM_DOCKERFILE=foo",
-		},
-	})
-	ctr, err := runEnvbuilder(t, runOpts{env: []string{
-		envbuilderEnv("GIT_URL", srv.URL),
-		envbuilderEnv("GIT_PASSWORD", "supersecret"),
-		options.EnvWithBuildSecretPrefix("FOO", "foo"),
-		envbuilderEnv("INIT_SCRIPT", "env > /root/env.txt && sleep infinity"),
-	}})
-	require.NoError(t, err)
-
-	output := execContainer(t, ctr, "cat /root/env.txt")
-	envsAvailableToInitScript := strings.Split(strings.TrimSpace(output), "\n")
-
-	leftoverBuildSecrets := options.GetBuildSecrets(envsAvailableToInitScript)
-	require.Empty(t, leftoverBuildSecrets, "build secrets should not be available to init script")
-}
-
 func TestBuildSecrets(t *testing.T) {
 	t.Parallel()
 
@@ -1175,7 +1145,7 @@ func TestBuildSecrets(t *testing.T) {
 	ctr, err := runEnvbuilder(t, runOpts{env: []string{
 		envbuilderEnv("GIT_URL", srv.URL),
 		envbuilderEnv("GIT_PASSWORD", "supersecret"),
-		options.EnvWithBuildSecretPrefix("FOO", buildSecretVal),
+		envbuilderEnv("BUILD_SECRETS", fmt.Sprintf("FOO=%s", buildSecretVal)),
 	}})
 	require.NoError(t, err)
 
diff --git a/options/secrets_test.go b/options/secrets_test.go
deleted file mode 100644
index 69193d9..0000000
--- a/options/secrets_test.go
+++ /dev/null
@@ -1,138 +0,0 @@
-package options_test
-
-import (
-	"os"
-	"strings"
-	"testing"
-
-	"github.com/coder/envbuilder/options"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-func TestGetBuildSecrets(t *testing.T) {
-	// This test cannot be run in parallel, because it needs to modify the OS environment
-	tests := []struct {
-		name            string
-		envVars         map[string]string
-		expectedSecrets []string
-	}{
-		{
-			name:            "no secrets set",
-			envVars:         map[string]string{},
-			expectedSecrets: []string{},
-		},
-		{
-			name: "single secret",
-			envVars: map[string]string{
-				"ENVBUILDER_BUILD_SECRET_FOO": "bar",
-			},
-			expectedSecrets: []string{"FOO=bar"},
-		},
-		{
-			name: "multiple secrets",
-			envVars: map[string]string{
-				"ENVBUILDER_BUILD_SECRET_FOO": "bar",
-				"NOT_A_SECRET":                "baz",
-				"ENVBUILDER_BUILD_SECRET_BAZ": "qux",
-			},
-			expectedSecrets: []string{"FOO=bar", "BAZ=qux"},
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			preserveEnv(t)
-			os.Clearenv()
-
-			options.ClearBuildSecretsFromProcessEnvironment()
-			require.Empty(t, options.GetBuildSecrets(os.Environ()))
-
-			// Set environment variables for the test case
-			for key, value := range tt.envVars {
-				t.Setenv(key, value)
-			}
-
-			// when
-			secrets := options.GetBuildSecrets(os.Environ())
-
-			// then
-			assert.ElementsMatch(t, tt.expectedSecrets, secrets)
-		})
-	}
-}
-
-func TestClearBuildSecrets(t *testing.T) {
-	// This test cannot be run in parallel, because it needs to modify the OS environment
-	tests := []struct {
-		name                       string
-		initialEnvVars             map[string]string
-		expectedSecretsBeforeClear []string
-		expectedEnvironAfterClear  []string
-	}{
-		{
-			name: "single secret",
-			initialEnvVars: map[string]string{
-				"ENVBUILDER_BUILD_SECRET_FOO": "bar",
-			},
-			expectedSecretsBeforeClear: []string{"FOO=bar"},
-		},
-		{
-			name: "multiple secrets",
-			initialEnvVars: map[string]string{
-				"ENVBUILDER_BUILD_SECRET_FOO": "bar",
-				"ENVBUILDER_BUILD_SECRET_BAZ": "qux",
-			},
-			expectedSecretsBeforeClear: []string{"FOO=bar", "BAZ=qux"},
-		},
-		{
-			name: "only build secrets are cleared",
-			initialEnvVars: map[string]string{
-				"ENVBUILDER_BUILD_SECRET_FOO": "foo",
-				"BAR":                         "bar",
-			},
-			expectedSecretsBeforeClear: []string{"FOO=foo"},
-			expectedEnvironAfterClear:  []string{"BAR=bar"},
-		},
-	}
-
-	for _, tt := range tests {
-		t.Run(tt.name, func(t *testing.T) {
-			preserveEnv(t)
-			os.Clearenv()
-
-			// Set environment variables for the test case
-			for key, value := range tt.initialEnvVars {
-				t.Setenv(key, value)
-			}
-
-			// Verify secrets before clearing
-			secrets := options.GetBuildSecrets(os.Environ())
-			assert.ElementsMatch(t, tt.expectedSecretsBeforeClear, secrets)
-
-			// Clear the secrets
-			options.ClearBuildSecretsFromProcessEnvironment()
-
-			// Verify secrets after clearing
-			environ := os.Environ()
-			secrets = options.GetBuildSecrets(environ)
-			assert.Empty(t, secrets)
-		})
-	}
-}
-
-// preserveEnv takes a snapshot of the current process environment and restores it after the current
-// test to ensure that we don't cause flakes by modifying the environment for other tests.
-func preserveEnv(t *testing.T) {
-	envSnapshot := make(map[string]string)
-	for _, envVar := range os.Environ() {
-		parts := strings.SplitN(envVar, "=", 2)
-		envSnapshot[parts[0]] = parts[1]
-	}
-	t.Cleanup(func() {
-		os.Clearenv()
-		for key, value := range envSnapshot {
-			os.Setenv(key, value)
-		}
-	})
-}
diff --git a/options/testdata/options.golden b/options/testdata/options.golden
index 397d5e0..840b234 100644
--- a/options/testdata/options.golden
+++ b/options/testdata/options.golden
@@ -12,6 +12,10 @@ OPTIONS:
           WorkspaceFolder. This path MUST be relative to the WorkspaceFolder
           path into which the repo is cloned.
 
+      --build-secrets string-array, $ENVBUILDER_BUILD_SECRETS
+          The list of secret environment variables to use when building the
+          image.
+
       --cache-repo string, $ENVBUILDER_CACHE_REPO
           The name of the container registry to push the cache image to. If this
           is empty, the cache will not be pushed.