Repository: comic/grand-challenge.org

Issue:
Importing of containers exported with Docker 25.0.0 fails
**Describe the bug**

I tried to import a viewer on grand-challenge right now, and it does not work:

![image](https://github.com/comic/grand-challenge.org/assets/11632379/f3684884-a283-4cbc-9697-e35323ceeaf6)

The failed images after the green, imported one are the ones I tried to upload:

- 1&2 are builds and clean re-builds of a dev branch
- 3 is a build of the CIRRUS main branch
- 4 is a known-good downloaded image from aws codebuild

None of them are importing right now, all fail with:

~~~~~~~~~~~~
Validation Errors
    blobs/sha256/e73667bff9013e92f65a7074744cbabffd2ce629e3da19adab4.json not found at the root of the container image file. Was this created with docker save?
~~~~~~~~~~~~

I manually opened the uploaded containers and the blobs are there. The files are also importing fine into docker. So I think that this time I have an actual grand-challenge issue at hand!

**To Reproduce**

Take a known-good image from aws-code build and try to upload to a viewer on grand-challenge.

**Expected behavior**

The image imports sucessfully.




Discussion:
Failures are logged here: https://grand-challenge.org/viewers/pkg-demo/

Ok... I think I found something:

![image](https://github.com/comic/grand-challenge.org/assets/11632379/0872b90b-eb2e-44d4-a611-d4e4bfa39d14)

I suspect that docker had some sort of update and exported blob-json files do not get the `.json` suffix attached anymore with a newer docker version... trying to confirm this now!
I think that I tracked down the code that _would_ need to update:

https://github.com/comic/grand-challenge.org/blob/7536b1e0361f0d9adf2b3afa0ecee3d9efe5c67e/app/grandchallenge/components/tasks.py#L451-L453

I now need to confirm that this is actually what needs to happen...
I can confirm that the manifest-format itself has changed over time/recently(?):

![image](https://github.com/comic/grand-challenge.org/assets/11632379/8eaa5c8a-8ea3-427c-8bf4-ad15420002d0)

The left side shows a freshly exported image, the right-hand side shows an old, stored image...
For references, I am using this docker version:

~~~~
paul@dingo-linux:~$ docker version 
Client: Docker Engine - Community
 Version:           25.0.0
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        e758fe5
 Built:             Thu Jan 18 17:09:49 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.0
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       615dfdf
  Built:            Thu Jan 18 17:09:49 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

~~~~
Aaand found the related issue that was fixed in version 25.0.0:

https://github.com/moby/moby/pull/44598

If I understand it correctly, the .json extension was stripped to be OCI-compliant
You are on it, James? I saw yout self-assignment. Then I stop my snails pace fixing adventure that I embarked on right now! :D
Yes should be an easy enough fix. You can work around it by using an older version of Docker but we should support the new stuff too.
> Yes should be an easy enough fix. You can work around it by using an older version of Docker but we should support the new stuff too.

Yes, big thanks! I will do some _local_ release testing CIRRUS for now then. That typically allows me to find obvious issues as well! If really necessary, downgrading docker is no problem either!
> Importing of containers **~built~ exported(!)** with Docker 25.0.0 fails

If you `docker save` an old image, the newer manifest gets written as well!

Resolving patch:
diff --git a/app/grandchallenge/components/tasks.py b/app/grandchallenge/components/tasks.py
--- a/app/grandchallenge/components/tasks.py
+++ b/app/grandchallenge/components/tasks.py
@@ -435,23 +435,10 @@ def _decompress_tarball(*, in_fileobj, out_fileobj):
 
 
 def _validate_docker_image_manifest(*, instance) -> str:
-    manifest = _extract_docker_image_file(
-        instance=instance, filename="manifest.json"
-    )
-    manifest = json.loads(manifest)
+    config_and_sha256 = _get_image_config_and_sha256(instance=instance)
 
-    if len(manifest) != 1:
-        raise ValidationError(
-            f"The container image file should only have 1 image. "
-            f"This file contains {len(manifest)}."
-        )
-
-    image_sha256 = manifest[0]["Config"][:64]
-
-    config = _extract_docker_image_file(
-        instance=instance, filename=f"{image_sha256}.json"
-    )
-    config = json.loads(config)
+    config = config_and_sha256["config"]
+    image_sha256 = config_and_sha256["image_sha256"]
 
     user = str(config["config"].get("User", "")).lower()
     if (
@@ -478,24 +465,84 @@ def _validate_docker_image_manifest(*, instance) -> str:
     return f"sha256:{image_sha256}"
 
 
-def _extract_docker_image_file(*, instance, filename: str):
-    """Extract a file from the root of a tarball."""
+def _get_image_config_and_sha256(*, instance):
     try:
         with instance.image.open(mode="rb") as im, tarfile.open(
             fileobj=im, mode="r"
-        ) as t:
-            member = dict(zip(t.getnames(), t.getmembers(), strict=True))[
-                filename
-            ]
-            file = t.extractfile(member).read()
-        return file
-    except (KeyError, tarfile.ReadError):
+        ) as open_tarfile:
+            container_image_files = {
+                tarinfo.name: tarinfo
+                for tarinfo in open_tarfile.getmembers()
+                if tarinfo.isfile()
+            }
+
+            image_manifest = _get_image_manifest(
+                container_image_files=container_image_files,
+                open_tarfile=open_tarfile,
+            )
+
+            return _get_image_config_file(
+                image_manifest=image_manifest,
+                container_image_files=container_image_files,
+                open_tarfile=open_tarfile,
+            )
+
+    except (EOFError, zlib.error, LZMAError, tarfile.ReadError, MemoryError):
+        raise ValidationError("Could not decompress the container image file.")
+
+
+def _get_image_manifest(*, container_image_files, open_tarfile):
+    try:
+        manifest = json.loads(
+            open_tarfile.extractfile(
+                container_image_files["manifest.json"]
+            ).read()
+        )
+    except KeyError:
         raise ValidationError(
-            f"{filename} not found at the root of the container image "
-            f"file. Was this created with docker save?"
+            "Could not find manifest.json in the container image file. "
+            "Was this created with docker save?"
         )
-    except (EOFError, zlib.error, LZMAError):
-        raise ValidationError("Could not decompress the container image file.")
+
+    if len(manifest) != 1:
+        raise ValidationError(
+            f"The container image file should only have 1 image. "
+            f"This file contains {len(manifest)}."
+        )
+
+    return manifest[0]
+
+
+def _get_image_config_file(
+    *, image_manifest, container_image_files, open_tarfile
+):
+    config_filename = image_manifest["Config"]
+
+    try:
+        config = json.loads(
+            open_tarfile.extractfile(
+                container_image_files[config_filename]
+            ).read()
+        )
+    except KeyError:
+        raise ValidationError(
+            "Could not find the config file in the container image file. "
+            "Was this created with docker save?"
+        )
+
+    if config_filename.endswith(".json"):
+        # Docker v2 container image
+        image_sha256 = config_filename.split(".")[0]
+    else:
+        # OCI container image
+        image_sha256 = image_manifest["Config"].split("/")[-1]
+
+    if len(image_sha256) != 64:
+        raise ValidationError(
+            "The container image file does not have a valid sha256 hash."
+        )
+
+    return {"image_sha256": image_sha256, "config": config}
 
 
 def retry_if_dropped(func):


Test patch:
diff --git a/app/tests/algorithms_tests/test_tasks.py b/app/tests/algorithms_tests/test_tasks.py
--- a/app/tests/algorithms_tests/test_tasks.py
+++ b/app/tests/algorithms_tests/test_tasks.py
@@ -144,7 +144,7 @@ def test_jobs_workflow(django_capture_on_commit_callbacks):
 
 @pytest.mark.django_db
 def test_algorithm(
-    client, algorithm_image, settings, django_capture_on_commit_callbacks
+    algorithm_image, settings, django_capture_on_commit_callbacks
 ):
     # Override the celery settings
     settings.task_eager_propagates = (True,)
diff --git a/app/tests/components_tests/resources/hello-scratch-docker-v2.tar.gz b/app/tests/components_tests/resources/hello-scratch-docker-v2.tar.gz
new file mode 100644
Binary files /dev/null and b/app/tests/components_tests/resources/hello-scratch-docker-v2.tar.gz differ
diff --git a/app/tests/components_tests/resources/hello-scratch-oci.tar.gz b/app/tests/components_tests/resources/hello-scratch-oci.tar.gz
new file mode 100644
Binary files /dev/null and b/app/tests/components_tests/resources/hello-scratch-oci.tar.gz differ
diff --git a/app/tests/components_tests/test_tasks.py b/app/tests/components_tests/test_tasks.py
--- a/app/tests/components_tests/test_tasks.py
+++ b/app/tests/components_tests/test_tasks.py
@@ -1,14 +1,17 @@
 import json
 import subprocess
+from pathlib import Path
 
 import pytest
 from celery.exceptions import MaxRetriesExceededError
+from django.core.files.base import ContentFile
 from requests import put
 
 from grandchallenge.algorithms.models import AlgorithmImage
 from grandchallenge.cases.models import RawImageUploadSession
 from grandchallenge.components.models import ComponentInterfaceValue
 from grandchallenge.components.tasks import (
+    _get_image_config_and_sha256,
     _repo_login_and_run,
     _retry,
     add_file_to_object,
@@ -422,3 +425,22 @@ def test_add_file_to_object(settings, object_type):
         interface_pk=ci.pk,
     )
     assert ComponentInterfaceValue.objects.filter(interface=ci).count() == 1
+
+
+@pytest.mark.parametrize(
+    "container_image_file",
+    ("hello-scratch-oci.tar.gz", "hello-scratch-docker-v2.tar.gz"),
+)
+@pytest.mark.django_db
+def test_get_image_config_and_sha256(container_image_file):
+    resource_dir = Path(__file__).parent / "resources"
+
+    ai = AlgorithmImageFactory(image=None)
+
+    with open(resource_dir / container_image_file, "rb") as f:
+        ai.image.save(container_image_file, ContentFile(f.read()))
+
+    assert (
+        _get_image_config_and_sha256(instance=ai)["image_sha256"]
+        == "1bf4ef3c617a6f34a728ec2a5cff1b1dcb926d2d0b93c5bccd830a7918d833da"
+    )
diff --git a/app/tests/evaluation_tests/test_tasks.py b/app/tests/evaluation_tests/test_tasks.py
--- a/app/tests/evaluation_tests/test_tasks.py
+++ b/app/tests/evaluation_tests/test_tasks.py
@@ -209,7 +209,7 @@ def test_method_validation_not_a_docker_tar(submission_file):
 
     method = Method.objects.get(pk=method.pk)
     assert method.is_manifest_valid is False
-    assert "manifest.json not found" in method.status
+    assert method.status == "Could not decompress the container image file."
 
 
 class TestSetEvaluationInputs(TestCase):