Repository: buildpacks/pack

Issue:
With pack 0.39.0, builds on arm64 fail with a problem pulling lifecycle
### Summary

When doing a build on AMD64, the build fails with this error:

```
            Builder 'paketobuildpacks/builder-jammy-full:latest' is untrusted
            As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
            For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
            Selected run image 'index.docker.io/paketobuildpacks/run-jammy-full:latest'
            Using lifecycle docker.io/buildpacksio/lifecycle:0.20.19; pulling digest index.docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5 for platform linux/amd64
            ERROR: failed to build: fetching lifecycle image: image 'index.docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5' does not exist on the daemon: not found
```

We are seeing this on AMD64 github runners and also on local AMD64 machines, but builds are ok on ARM64 Macbooks.

---

### Reproduction

##### Steps

We've seen it with a couple different commands:

```
pack build test123 --no-color \
--path  ./paketo_samples/nodejs/npm --buildpack docker.io/paketobuildpacks/node-engine \
--buildpack docker.io/paketobuildpacks/npm-install \
--buildpack docker.io/paketobuildpacks/npm-start
```

### Environment

##### pack info

pack 0.39.0

##### docker info

It looks like our runner is 24.04, so it should have Docker 28, according to [this](https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md).

We are seeing this on a laptop with Docker version 29.0.4, build 3247a5a


Discussion:
@dmikusa: Updated steps to reproduce:

1. `docker pull docker.io/paketobuildpacks/builder-jammy-full:latest`
2. `docker pull docker.io/buildpacksio/lifecycle:0.20.19`
3. `docker pull docker.io/paketobuildpacks/run-jammy-full:latest`
4. `pack build -v --pull-policy never --builder docker.io/paketobuildpacks/builder-jammy-full:latest -e BP_TOMCAT_VERSION=10.1 apps/war-spring` (uses https://github.com/paketo-buildpacks/samples/tree/main/java/war-spring)

```
> ../../pack build -v --pull-policy never --builder docker.io/paketobuildpacks/builder-jammy-full:latest -e BP_TOMCAT_VERSION=10.1 apps/war-spring
Builder docker.io/paketobuildpacks/builder-jammy-full:latest is untrusted
As a result, the phases of the lifecycle which require root access will be run in separate trusted ephemeral containers.
For more information, see https://medium.com/buildpacks/faster-more-secure-builds-with-pack-0-11-0-4d0c633ca619
'Selected run image index.docker.io/paketobuildpacks/run-jammy-full:latest
Using lifecycle docker.io/buildpacksio/lifecycle:0.20.19; pulling digest index.docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5 for platform linux/amd64
ERROR: failed to build: fetching lifecycle image: image index.docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5 does not exist on the daemon: not found
```

```
> docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
paketobuildpacks/run-jammy-full       latest    a82dcffabd38   34 hours ago   715MB
buildpacksio/lifecycle                0.20.19   834568335a3f   45 years ago   29.6MB
paketobuildpacks/builder-jammy-full   latest    813215361121   45 years ago   4GB
```

```
> docker inspect buildpacksio/lifecycle:0.20.19
[
    {
        "Id": "sha256:834568335a3fc750beee8ef72c2c18b9a9cdde7890c26fbccce5d72575cee5a3",
        "RepoTags": [
            "buildpacksio/lifecycle:0.20.19"
        ],
        "RepoDigests": [
            "buildpacksio/lifecycle@sha256:85b41ff7170c517a464279a5ba9fc4ce7e4ffc42abf99b149175808b5fe753ab"
        ],
        "Parent": "",
        "Comment": "",
        "Created": "1980-01-01T00:00:01Z",
        "DockerVersion": "",
        "Author": "",
        "Architecture": "amd64",
        "Os": "linux",
        "Size": 29556398,
        "GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/1afeefb914592dbe9eafc9ccf5f5ef549fe9b4427644eceb310bf901d17294c2/diff:/var/lib/docker/overlay2/e4151502b9c8e55f33a5b305df11c8206bb8b1aebfaa98b5b8421a8b09451bd1/diff:/var/lib/docker/overlay2/734058b47ad7727e6107fdee506072b1276bdacead4b43d1d9524e82ea42fce1/diff:/var/lib/docker/overlay2/84c8c5d20c53f83886a489eff003a2147bef03ec078c5bc0b8d4667a5e29e856/diff:/var/lib/docker/overlay2/348df328b3d70a5279a818d4135e8ff865df3abb9ae2653e388321a5c5282050/diff:/var/lib/docker/overlay2/6ca04faf5b1dc25947d881368ef91c437ed3398f716a27facddad4452bce8416/diff:/var/lib/docker/overlay2/5317d29cab0780727472835e85052c08e894b8b7f765600b8f63104422b71c9b/diff:/var/lib/docker/overlay2/0773556c7fd2ba965918db59136c89d671cc008c9ada92702080a7f1f2b7bcb8/diff:/var/lib/docker/overlay2/4ac186c52d32478f75ad5650e3dc1f8090a9efa9f7bd4a6e8666e95dc83e81e9/diff:/var/lib/docker/overlay2/c94ce5d49c55063394898b103c8aec2ec18947b775afe10e65c1e702cc9d4044/diff:/var/lib/docker/overlay2/0d6a72108f9e3ab81450eb49b7fda733b8d760124ef5971b6f205412cc76d3d2/diff:/var/lib/docker/overlay2/d1a90391dd8ea4de359a4f273e0879ddaf171b410dd633f0d6f03f4e4b7895ad/diff",
                "MergedDir": "/var/lib/docker/overlay2/b54a7fa287869c631de920434a0f5f57c12f62484f3bde08c4c3ab337744f892/merged",
                "UpperDir": "/var/lib/docker/overlay2/b54a7fa287869c631de920434a0f5f57c12f62484f3bde08c4c3ab337744f892/diff",
                "WorkDir": "/var/lib/docker/overlay2/b54a7fa287869c631de920434a0f5f57c12f62484f3bde08c4c3ab337744f892/work"
            },
            "Name": "overlay2"
        },
        "RootFS": {
            "Type": "layers",
            "Layers": [
                "sha256:bff7f7a9d44356d8784500366094c66399aa6a2edd990cc70e02e27c84402753",
                "sha256:8fa10c0194df9b7c054c90dbe482585f768a54428fc90a5b78a0066a123b1bba",
                "sha256:4840c7c54023c867f19564429c89ddae4e9589c83dce82492183a7e9f7dab1fa",
                "sha256:114dde0fefebbca13165d0da9c500a66190e497a82a53dcaabc3172d630be1e9",
                "sha256:4d049f83d9cf21d1f5cc0e11deaf36df02790d0e60c1a3829538fb4b61685368",
                "sha256:af5aa97ebe6ce1604747ec1e21af7136ded391bcabe4acef882e718a87c86bcc",
                "sha256:6f1cdceb6a3146f0ccb986521156bef8a422cdbb0863396f7f751f575ba308f4",
                "sha256:bbb6cacb8c82e4da4e8143e03351e939eab5e21ce0ef333c42e637af86c5217b",
                "sha256:2a92d6ac9e4fcc274d5168b217ca4458a9fec6f094ead68d99c77073f08caac1",
                "sha256:1a73b54f556b477f0a8b939d13c504a3b4f4db71f7a09c63afbc10acb3de5849",
                "sha256:f4aee9e53c42a22ed82451218c3ea03d1eea8d6ca8fbe8eb4e950304ba8a8bb3",
                "sha256:bfe9137a1b044e8097cdfcb6899137a8a984ed70931ed1e8ef0cf7e023a139fc",
                "sha256:cb956c2f2a0f03ce5919776995889b72982e3c4dc8e560c6c09a1f3986c6c53f"
            ]
        },
        "Metadata": {
            "LastTagTime": "0001-01-01T00:00:00Z"
        },
        "Config": {
            "Cmd": null,
            "Entrypoint": null,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt"
            ],
            "Labels": {
                "io.buildpacks.builder.metadata": "{\"lifecycle\":{\"version\":\"0.20.19\"},\"api\":{\"buildpack\":\"0.7\",\"platform\":\"0.7\"}}",
                "io.buildpacks.lifecycle.apis": "{\"buildpack\":{\"deprecated\":[],\"supported\":[\"0.7\",\"0.8\",\"0.9\",\"0.10\",\"0.11\"]},\"platform\":{\"deprecated\":[],\"supported\":[\"0.7\",\"0.8\",\"0.9\",\"0.10\",\"0.11\",\"0.12\",\"0.13\",\"0.14\"]}}",
                "io.buildpacks.lifecycle.version": "0.20.19"
            },
            "OnBuild": null,
            "User": "0",
            "Volumes": null,
            "WorkingDir": "/layers"
        }
    }
]
```@dmikusa: Not 100% sure what is going on, but this is weird.

When it's not working:

```
> docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
paketobuildpacks/run-jammy-full       latest    a82dcffabd38   34 hours ago   715MB
paketobuildpacks/builder-jammy-full   latest    813215361121   45 years ago   4GB
buildpacksio/lifecycle                0.20.19   834568335a3f   45 years ago   29.6MB
```

If I pull the specific hash:

```
> docker pull docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5
docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5: Pulling from buildpacksio/lifecycle
Digest: sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5
Status: Downloaded newer image for buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5
docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5
```

It is actually pulling something down based on the output, but there is no change to docker images.

```
> docker images
REPOSITORY                            TAG       IMAGE ID       CREATED        SIZE
paketobuildpacks/run-jammy-full       latest    a82dcffabd38   34 hours ago   715MB
buildpacksio/lifecycle                0.20.19   834568335a3f   45 years ago   29.6MB
paketobuildpacks/builder-jammy-full   latest    813215361121   45 years ago   4GB
```

However, after that, `pack build` works. So clearly there is some layer that's missing when you `docker pull` the version tag that `pack` needs now.@dmikusa: It looks like when you `docker pull docker.io/buildpacksio/lifecycle:0.20.19`, you don't get `docker.io/buildpacksio/lifecycle@sha256:e49aa0fbf057babb6f4dc4bb8aa5c2c9fd202e398b51c706cc845f1c2affe7b5` which is a manifest blob, and `pack` is now looking for this manifest blob.

Since we set `--pull-policy never`, pack can't go and fetch that missing manifest so it fails.

Not sure if this is necessarily a bug or just a caveat of the way that we're running this, but I do suspect other users might hit this as I've seen people run with `--pull-policy never` as a way to control what pack is pulling from the Internet, in more restrictive environments.

Thoughts?

Solution patch:
diff --git a/pkg/image/fetcher.go b/pkg/image/fetcher.go
index ff9b2d18..e95fe796 100644
--- a/pkg/image/fetcher.go
+++ b/pkg/image/fetcher.go
@@ -257,6 +257,17 @@ func (f *Fetcher) FetchForPlatform(ctx context.Context, name string, options Fet
 		return nil, err
 	}
 
+	platformStr := options.Target.ValuesAsPlatform()
+
+	// When PullPolicy is PullNever, skip platform-specific digest resolution as it requires
+	// network access to fetch the manifest list. Instead, use the image as-is from the daemon.
+	// Note: This may cause issues with containerd storage. Users should pre-pull the platform-specific
+	// digest if they encounter errors.
+	if options.Daemon && options.PullPolicy == PullNever {
+		f.logger.Debugf("Using lifecycle %s with platform %s (skipping digest resolution due to --pull-policy never)", name, platformStr)
+		return f.Fetch(ctx, name, options)
+	}
+
 	// Build platform and registry settings from options
 	platform := imgutil.Platform{
 		OS:           options.Target.OS,
@@ -275,7 +286,6 @@ func (f *Fetcher) FetchForPlatform(ctx context.Context, name string, options Fet
 	}
 
 	// Log the resolution for visibility
-	platformStr := options.Target.ValuesAsPlatform()
 	f.logger.Debugf("Using lifecycle %s; pulling digest %s for platform %s", name, resolvedName, platformStr)
 
 	return f.Fetch(ctx, resolvedName, options)



Test patch:
diff --git a/pkg/image/fetcher_test.go b/pkg/image/fetcher_test.go
index 012914e0..2e6ff7c2 100644
--- a/pkg/image/fetcher_test.go
+++ b/pkg/image/fetcher_test.go
@@ -732,6 +732,49 @@ func testFetcher(t *testing.T, when spec.G, it spec.S) {
 				h.AssertError(t, err, "")
 			})
 		})
+
+		when("pull policy is PullNever with daemon", func() {
+			var localImageName string
+
+			it.Before(func() {
+				// Use a different name for the local image to avoid conflicts
+				localImageName = "pack.local/test-" + h.RandString(10)
+
+				// Create a local daemon image with platform information
+				// Use osType (daemon OS) instead of runtime.GOOS to handle cases where
+				// Windows runner is running Linux containers
+				img, err := local.NewImage(localImageName, docker, local.WithDefaultPlatform(imgutil.Platform{
+					OS:           osType,
+					Architecture: runtime.GOARCH,
+				}))
+				h.AssertNil(t, err)
+				h.AssertNil(t, img.Save())
+			})
+
+			it.After(func() {
+				h.DockerRmi(docker, localImageName)
+			})
+
+			it("skips platform-specific digest resolution and uses tag directly", func() {
+				target := dist.Target{
+					OS:   osType,
+					Arch: runtime.GOARCH,
+				}
+
+				fetchedImg, err := imageFetcher.FetchForPlatform(context.TODO(), localImageName, image.FetchOptions{
+					Daemon:     true,
+					PullPolicy: image.PullNever,
+					Target:     &target,
+				})
+
+				// Should succeed without network access (digest resolution skipped)
+				h.AssertNil(t, err)
+				h.AssertNotNil(t, fetchedImg)
+
+				// Verify debug message about skipping digest resolution
+				h.AssertContains(t, outBuf.String(), "skipping digest resolution due to --pull-policy never")
+			})
+		})
 	})
 
 	when("#CheckReadAccess", func() {