Repository: containers/image
Pull request #2695 (2025-01-23 17:48:48)

Add docker media types in OCI formats
This pull request adds the docker media types when reading back OCI formats, along with the corresponding tests.  This is from running into cases where docker images were built with the docker media types.  To avoid having to convert all of the existing manifests so that the original types can be maintained, this code adds support for reading the docker specific formats since the OCI formats are backwards compatible with the docker formats (https://github.com/opencontainers/image-spec/blob/main/media-types.md).

mtrmac: Thanks. At a high level, I’m really unsure.

For reading images, sure, that’s ~easy enough. But are the users then going to ask to also supporting _writes_ with non-OCI images? Does the `DeleteImage` code need to be able to handle non-OCI media types?

Approximately, I generally think that any producer creating an OCI-layout directory structure is obviously aware of OCI, so I don’t know why any _new_ user of this code should start to want to support pre-OCI images.

What’s the use case? 

If the goal were to write some kind of general-purpose image storage mechanism, note that the OCI transport (currently) can’t do that, e.g. it doesn’t support reading/writing signatures along with the image.
brbayes-msft: The main issue I'm looking to address here is cases where tooling such as the Docker Daemon pull existing images from upstreams, and those upstream images still use docker media types.  With the recent switch for the Docker Daemon to save dual format archives (both oci and docker archive), we run into an issue where the archive is still an OCI layout, but it maintained its original media types, thus breaking tools like Skopeo.

I do agree that when creating OCI layouts, the creator is likely aware of the OCI types and should conform to them when possible.  That's why I focused primarily on only read operations so as to better handle the case where tooling changes the wrapper.  That being said, if there are other areas of the code that I should fill out, I'm happy to do so.
brbayes-msft: @mtrmac Sorry to ping you, but I'd like to follow up here if there is a path forward to supporting the docker media types.
mtrmac: I’m sorry, I couldn’t return to look into this in detail — but, the short version of it, is that if `docker save` (or something similar?) produces such archives, then I do agree that it is valuable to be able to consume them.
brbayes-msft: @mtrmac Sorry to bump this again.  I updated the PR after your comments on V2S1.  Let me know if there's any other changes I should add.
mtrmac: > With the recent switch for the Docker Daemon to save dual format archives (both oci and docker archive), we run into an issue where the archive is still an OCI layout, but it maintained its original media types, thus breaking tools like Skopeo.

I can’t reproduce this:
```console
# rpm -q moby-engine skopeo
moby-engine-27.3.1-2.fc41.x86_64
skopeo-1.17.0-1.fc41.x86_64
# skopeo inspect docker://quay.io/libpod/alpine | jq '.Digest,.LayersData[].MIMEType' 
"sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f"
"application/vnd.docker.image.rootfs.diff.tar.gzip"
# docker pull quay.io/libpod/alpine
…
# docker save -o foo.tar quay.io/libpod/alpine
# skopeo inspect --raw oci-archive:foo.tar | jq -r .mediaType
application/vnd.oci.image.manifest.v1+json
```
(and extracting the archive, `index.json` uses OCI MIME types throughout).

Is there some other way this can be created using Docker?
brbayes-msft: > > With the recent switch for the Docker Daemon to save dual format archives (both oci and docker archive), we run into an issue where the archive is still an OCI layout, but it maintained its original media types, thus breaking tools like Skopeo.
> 
> I can’t reproduce this:
> 
> ```
> # rpm -q moby-engine skopeo
> moby-engine-27.3.1-2.fc41.x86_64
> skopeo-1.17.0-1.fc41.x86_64
> # skopeo inspect docker://quay.io/libpod/alpine | jq '.Digest,.LayersData[].MIMEType' 
> "sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f"
> "application/vnd.docker.image.rootfs.diff.tar.gzip"
> # docker pull quay.io/libpod/alpine
> …
> # docker save -o foo.tar quay.io/libpod/alpine
> # skopeo inspect --raw oci-archive:foo.tar | jq -r .mediaType
> application/vnd.oci.image.manifest.v1+json
> ```
> 
> (and extracting the archive, `index.json` uses OCI MIME types throughout).
> 
> Is there some other way this can be created using Docker?

I see different behavior on my machine.  I run Windows, though we have run into this on Linux hosts as well.

``` powershell
PS C:\Users\brbayes\Downloads> docker -v
Docker version 27.5.1, build 9f9e405
PS C:\Users\brbayes\Downloads> skopeo -v
skopeo version 1.17.0
PS C:\Users\brbayes\Downloads> skopeo inspect docker://quay.io/libpod/alpine --override-os linux | jq '.Digest,.LayersData[].MIMEType'
"sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f"
"application/vnd.docker.image.rootfs.diff.tar.gzip"
PS C:\Users\brbayes\Downloads> docker pull quay.io/libpod/alpine
Using default tag: latest
latest: Pulling from libpod/alpine
Digest: sha256:fa93b01658e3a5a1686dc3ae55f170d8de487006fb53a28efcd12ab0710a2e5f
Status: Image is up to date for quay.io/libpod/alpine:latest
quay.io/libpod/alpine:latest

What's next:
    View a summary of image vulnerabilities and recommendations → docker scout quickview quay.io/libpod/alpine
PS C:\Users\brbayes\Downloads> docker save -o libpod.tar quay.io/libpod/alpine
PS C:\Users\brbayes\Downloads> skopeo inspect --raw oci-archive:libpod.tar | jq -r .mediaType
application/vnd.docker.distribution.manifest.list.v2+json
```
mtrmac: Thanks.

I’m looking at https://github.com/moby/moby/blob/8ca767963101763af2d8e44ea24ae0756adcfb05/image/tarexport/save.go#L250, that’s clearly not that path.

This is with the containerd backend, isn’t it?
brbayes-msft: That's correct, this is using the containerd backend both on my local machine, and on the linux hosts.
mtrmac: Thanks, reproduced. Fine, let’s do this.

… and it saves the original multi-platform manifest list, but the archive only contains the one pulled architecture. I.e. the image is sparse. That’s going to be fun.

Patch:
diff --git a/oci/layout/fixtures/name_lookups/index.json b/oci/layout/fixtures/name_lookups/index.json
index b79d1089..5af85e7f 100644
--- a/oci/layout/fixtures/name_lookups/index.json
+++ b/oci/layout/fixtures/name_lookups/index.json
@@ -3,7 +3,7 @@
     "mediaType": "application/vnd.oci.image.index.v1+json",
     "manifests": [
         {
-            "mediaType": "application/vnd.oci.image.manifest.v1+json",
+            "mediaType": "application/vnd.oci.image.index.v1+json",
             "digest": "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
             "size": 1,
             "annotations": {
@@ -19,17 +19,25 @@
             }
         },
         {
-            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
+            "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json",
             "digest": "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
             "size": 3,
             "annotations": {
-                "org.opencontainers.image.ref.name": "invalid-mime"
+                "org.opencontainers.image.ref.name": "c"
             }
         },
         {
-            "mediaType": "x-completely-unknown",
+            "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
             "digest": "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
             "size": 4,
+            "annotations": {
+                "org.opencontainers.image.ref.name": "d"
+            }
+        },
+        {
+            "mediaType": "x-completely-unknown",
+            "digest": "sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
+            "size": 6,
             "annotations": {
                 "org.opencontainers.image.ref.name": "invalid-mime"
             }
diff --git a/oci/layout/oci_transport.go b/oci/layout/oci_transport.go
index 6ab38467..800d1aa2 100644
--- a/oci/layout/oci_transport.go
+++ b/oci/layout/oci_transport.go
@@ -12,6 +12,7 @@ import (
 	"github.com/containers/image/v5/directory/explicitfilepath"
 	"github.com/containers/image/v5/docker/reference"
 	"github.com/containers/image/v5/internal/image"
+	"github.com/containers/image/v5/internal/manifest"
 	"github.com/containers/image/v5/oci/internal"
 	"github.com/containers/image/v5/transports"
 	"github.com/containers/image/v5/types"
@@ -234,7 +235,7 @@ func (ref ociReference) getManifestDescriptor() (imgspecv1.Descriptor, int, erro
 		var unsupportedMIMETypes []string
 		for i, md := range index.Manifests {
 			if refName, ok := md.Annotations[imgspecv1.AnnotationRefName]; ok && refName == ref.image {
-				if md.MediaType == imgspecv1.MediaTypeImageManifest || md.MediaType == imgspecv1.MediaTypeImageIndex {
+				if md.MediaType == imgspecv1.MediaTypeImageManifest || md.MediaType == imgspecv1.MediaTypeImageIndex || md.MediaType == manifest.DockerV2Schema2MediaType || md.MediaType == manifest.DockerV2ListMediaType {
 					return md, i, nil
 				}
 				unsupportedMIMETypes = append(unsupportedMIMETypes, md.MediaType)


Test patch:
diff --git a/oci/layout/oci_transport_test.go b/oci/layout/oci_transport_test.go
index 5c608737..5fb7b2ef 100644
--- a/oci/layout/oci_transport_test.go
+++ b/oci/layout/oci_transport_test.go
@@ -53,7 +53,7 @@ func TestGetManifestDescriptor(t *testing.T) {
 			dir:   "fixtures/name_lookups",
 			image: "a",
 			expectedDescriptor: &imgspecv1.Descriptor{
-				MediaType:   "application/vnd.oci.image.manifest.v1+json",
+				MediaType:   "application/vnd.oci.image.index.v1+json",
 				Digest:      "sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
 				Size:        1,
 				Annotations: map[string]string{"org.opencontainers.image.ref.name": "a"},
@@ -71,6 +71,28 @@ func TestGetManifestDescriptor(t *testing.T) {
 			},
 			expectedIndex: 1,
 		},
+		{ // A valid reference in a multi-manifest directory
+			dir:   "fixtures/name_lookups",
+			image: "c",
+			expectedDescriptor: &imgspecv1.Descriptor{
+				MediaType:   "application/vnd.docker.distribution.manifest.list.v2+json",
+				Digest:      "sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
+				Size:        3,
+				Annotations: map[string]string{"org.opencontainers.image.ref.name": "c"},
+			},
+			expectedIndex: 2,
+		},
+		{ // A valid reference in a multi-manifest directory
+			dir:   "fixtures/name_lookups",
+			image: "d",
+			expectedDescriptor: &imgspecv1.Descriptor{
+				MediaType:   "application/vnd.docker.distribution.manifest.v2+json",
+				Digest:      "sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
+				Size:        4,
+				Annotations: map[string]string{"org.opencontainers.image.ref.name": "d"},
+			},
+			expectedIndex: 3,
+		},
 		{ // No entry found
 			dir:                "fixtures/name_lookups",
 			image:              "this-does-not-exist",
diff --git a/oci/layout/reader_test.go b/oci/layout/reader_test.go
index d59c281b..3d964c3e 100644
--- a/oci/layout/reader_test.go
+++ b/oci/layout/reader_test.go
@@ -35,14 +35,15 @@ func TestList(t *testing.T) {
 		},
 		{
 			path: "fixtures/name_lookups",
-			num:  4,
+			num:  5,
 			digests: []string{
 				"sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
 				"sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
 				"sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
 				"sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
+				"sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
 			},
-			names: map[int]string{0: "a", 1: "b", 2: "invalid-mime", 3: "invalid-mime"},
+			names: map[int]string{0: "a", 1: "b", 2: "c", 3: "d", 4: "invalid-mime"},
 		},
 	} {
 		results, err := List(test.path)