Repository: comic/grand-challenge.org

Issue:
TIFF files are invalid as (new) segmentation values
Validation always fails for TIFF files when uploading as a component-interface value with sub kind segmentation. (When `overlay_segments` are set).

Creating new segmentation component interfaces forces the addition of the `overlay_segments`. However, TIFF files have no value range calculated as their typical size (~4.000.000.000 voxels) makes it impractical to calculate v.s. regular MHA files (~200.000.000) as such the `image.segments` is never filled and the validation will always fail.




Discussion:
It should be investigated if panimg can retrieve the values present quickly or not.
It all depends on the image. We usually deal with multi-resolution images, but that is not always the case. A tiff file can also be a single resolution, single page tiff file of 4 GB. Loading that and getting the unique voxel values from such an image requires a lot of memory. We do extract the min/max value from a tiff file, so maybe that's a good compromise? If you want a segmentation in a tiff file, the overlag segments need to have consecutive values defined, and we just validate on the min/max. 
Good idea, for that check we would also need to make sure:

- The overlay segments that are defined for the interface are contiguous (e.g. {1, 2, 3} is OK for that check, {10, 20, 30} is not)
- The min of the tiff file is >= the min of the expected overlay segments
- The max of the tiff file is <= the max of the expected overlay segments
- The data type of the tiff file is `int` so that only the integers between the defined min and max value are possible

If that is possible then that should work for most cases.
Yes, I think that is the most efficient solution, and a lot of segmentations have contiguous values anyway. Should I make a pitch or do you think it's too little work to justify a pitch? 
Famous last words but I don't think it's that much work so a PR would be fine.
`libvips` can aggregate that kind of information rather quickly in my experience. It is a pretty awesome library, however, using it correctly sometimes requires some "esoteric" constructs.

Also: If we create this kind of information, we can even maybe decide to build png-encoded DZI files depending on the complexity of the data that is being encoded. That will solve a big issue "blurry"/incorrect class overlays ins Pathology images.
> Good idea, for that check we would also need to make sure:
> 
>     * The overlay segments that are defined for the interface are contiguous (e.g. {1, 2, 3} is OK for that check, {10, 20, 30} is not)
> 
>     * The min of the tiff file is >= the min of the expected overlay segments
> 
>     * The max of the tiff file is <= the max of the expected overlay segments
> 
>     * The data type of the tiff file is `int` so that only the integers between the defined min and max value are possible
> 
> 
> If that is possible then that should work for most cases.

We do this only for tiff I would say? So we keep the option for non-contiguous overlay segments available for other modalities.
We should check the existing interfaces to see if there are any non contiguous overlays, if not then I think that it would be fine to add as a constraint.
There are overlay segments now that are not contiguous.
OK then yes we would not be able to validate TIFF files for those ones, so raise a validation error there.

Resolving patch:
diff --git a/app/grandchallenge/components/models.py b/app/grandchallenge/components/models.py
--- a/app/grandchallenge/components/models.py
+++ b/app/grandchallenge/components/models.py
@@ -661,32 +661,42 @@ class OverlaySegmentsMixin(models.Model):
     )
 
     @property
-    def voxel_values(self):
+    def overlay_segments_allowed_values(self):
         allowed_values = {x["voxel_value"] for x in self.overlay_segments}
-
         # An implicit background value of 0 is always allowed, this saves the
         # user having to declare it and the annotator mark it
         allowed_values.add(0)
 
         return allowed_values
 
+    @property
+    def overlay_segments_is_contiguous(self):
+        values = sorted(list(self.overlay_segments_allowed_values))
+        return all(
+            values[i] - values[i - 1] == 1 for i in range(1, len(values))
+        )
+
     def _validate_voxel_values(self, image):
         if not self.overlay_segments:
             return
 
         if image.segments is None:
             raise ValidationError(
-                "Image segments could not be determined, ensure the file is "
-                "not a tiff file, its voxel values are integers and that it "
-                f"contains no more than {MAXIMUM_SEGMENTS_LENGTH} segments."
+                "Image segments could not be determined, ensure the voxel values "
+                "are integers and that it contains no more than "
+                f"{MAXIMUM_SEGMENTS_LENGTH} segments. Ensure the image has the "
+                "minimum and maximum voxel values set as tags if the image is a TIFF "
+                "file."
             )
 
-        invalid_values = set(image.segments) - self.voxel_values
+        invalid_values = (
+            set(image.segments) - self.overlay_segments_allowed_values
+        )
         if invalid_values:
             raise ValidationError(
                 f"The valid voxel values for this segmentation are: "
-                f"{self.voxel_values}. This segmentation is invalid as "
-                f"it contains the voxel values: {invalid_values}."
+                f"{self.overlay_segments_allowed_values}. This segmentation is "
+                f"invalid as it contains the voxel values: {invalid_values}."
             )
 
     class Meta:
@@ -902,6 +912,11 @@ def _clean_overlay_segments(self):
                 "Overlay segments should only be set for segmentations"
             )
 
+        if not self.overlay_segments_is_contiguous:
+            raise ValidationError(
+                "Voxel values for overlay segments must be contiguous."
+            )
+
         Question = apps.get_model("reader_studies", "question")  # noqa: N806
         if (
             self.pk is not None


Test patch:
diff --git a/app/tests/cases_tests/factories.py b/app/tests/cases_tests/factories.py
--- a/app/tests/cases_tests/factories.py
+++ b/app/tests/cases_tests/factories.py
@@ -3,7 +3,7 @@
 import factory
 from factory import fuzzy
 
-from grandchallenge.cases.models import Image, RawImageUploadSession
+from grandchallenge.cases.models import Image, ImageFile, RawImageUploadSession
 from tests.cases_tests import RESOURCE_PATH
 from tests.factories import (
     ImageFactory,
@@ -50,6 +50,11 @@ class ImageFileFactoryWithMHA16Bit(ImageFileFactory):
     file = factory.django.FileField(from_path=RESOURCE_PATH / "image16bit.mha")
 
 
+class ImageFileFactoryWithTiff(ImageFileFactory):
+    file = factory.django.FileField(from_path=RESOURCE_PATH / "valid_tiff.tif")
+    image_type = ImageFile.IMAGE_TYPE_TIFF
+
+
 class ImageFactoryWithoutImageFile(ImageFactory):
     eye_choice = factory.Iterator([x[0] for x in Image.EYE_CHOICES])
     stereoscopic_choice = factory.Iterator(
@@ -154,6 +159,19 @@ def files(self, create, extracted, **kwargs):
             ImageFileFactoryWithMHA16Bit(image=self)
 
 
+class ImageFactoryWithImageFileTiff(ImageFactoryWithoutImageFile):
+    @factory.post_generation
+    def files(self, create, extracted, **kwargs):
+        # See https://factoryboy.readthedocs.io/en/latest/recipes.html#simple-many-to-many-relationship
+        if not create:
+            return
+        if extracted:
+            for image in extracted:
+                self.files.add(image)
+        if create and not extracted:
+            ImageFileFactoryWithTiff(image=self)
+
+
 class RawImageUploadSessionFactory(factory.django.DjangoModelFactory):
     class Meta:
         model = RawImageUploadSession
diff --git a/app/tests/components_tests/test_models.py b/app/tests/components_tests/test_models.py
--- a/app/tests/components_tests/test_models.py
+++ b/app/tests/components_tests/test_models.py
@@ -28,6 +28,7 @@
     AlgorithmImageFactory,
     AlgorithmJobFactory,
 )
+from tests.cases_tests.factories import ImageFactoryWithImageFileTiff
 from tests.components_tests.factories import (
     ComponentInterfaceFactory,
     ComponentInterfaceValueFactory,
@@ -1075,6 +1076,40 @@ def test_clean_overlay_segments_with_questions(reader_study_with_gt):
     )
 
 
+@pytest.mark.django_db
+def test_clean_overlay_segments():
+    ci = ComponentInterface(kind=InterfaceKindChoices.STRING)
+    ci.overlay_segments = [{"name": "s1", "visible": True, "voxel_value": 1}]
+    with pytest.raises(ValidationError) as e:
+        ci._clean_overlay_segments()
+    assert (
+        e.value.message
+        == "Overlay segments should only be set for segmentations"
+    )
+
+    ci = ComponentInterface(kind=InterfaceKindChoices.SEGMENTATION)
+    with pytest.raises(ValidationError) as e:
+        ci._clean_overlay_segments()
+    assert e.value.message == "Overlay segments must be set for this interface"
+
+    ci.overlay_segments = [
+        {"name": "s1", "visible": True, "voxel_value": 1},
+        {"name": "s2", "visible": True, "voxel_value": 3},
+    ]
+    with pytest.raises(ValidationError) as e:
+        ci._clean_overlay_segments()
+    assert (
+        e.value.message
+        == "Voxel values for overlay segments must be contiguous."
+    )
+
+    ci.overlay_segments = [
+        {"name": "s1", "visible": True, "voxel_value": 1},
+        {"name": "s2", "visible": True, "voxel_value": 2},
+    ]
+    ci._clean_overlay_segments()
+
+
 @pytest.mark.django_db
 def test_validate_voxel_values():
     ci = ComponentInterfaceFactory(kind=InterfaceKindChoices.SEGMENTATION)
@@ -1085,9 +1120,11 @@ def test_validate_voxel_values():
     ci.save()
 
     error_msg = (
-        "Image segments could not be determined, ensure the file is "
-        "not a tiff file, its voxel values are integers and that it "
-        f"contains no more than {MAXIMUM_SEGMENTS_LENGTH} segments."
+        "Image segments could not be determined, ensure the voxel values "
+        "are integers and that it contains no more than "
+        f"{MAXIMUM_SEGMENTS_LENGTH} segments. Ensure the image has the "
+        "minimum and maximum voxel values set as tags if the image is a TIFF "
+        "file."
     )
     im = ImageFactory(segments=None)
     with pytest.raises(ValidationError) as e:
@@ -1110,6 +1147,38 @@ def test_validate_voxel_values():
     im = ImageFactory(segments=[0, 1, 2])
     assert ci._validate_voxel_values(im) is None
 
+    im = ImageFactoryWithImageFileTiff()
+    ci.overlay_segments = [
+        {"name": "s1", "visible": True, "voxel_value": 1},
+        {"name": "s2", "visible": True, "voxel_value": 2},
+    ]
+    with pytest.raises(ValidationError) as e:
+        ci._validate_voxel_values(im)
+    assert e.value.message == error_msg
+    im = ImageFactoryWithImageFileTiff(segments=[1, 2, 3])
+    ci.overlay_segments = [
+        {"name": "s1", "visible": True, "voxel_value": 1},
+        {"name": "s2", "visible": True, "voxel_value": 3},
+    ]
+    with pytest.raises(ValidationError) as e:
+        ci._validate_voxel_values(im)
+    assert e.value.message == (
+        "The valid voxel values for this segmentation are: {0, 1, 3}. "
+        "This segmentation is invalid as it contains the voxel values: {2}."
+    )
+    ci.overlay_segments = [
+        {"name": "s1", "visible": True, "voxel_value": 1},
+        {"name": "s2", "visible": True, "voxel_value": 2},
+    ]
+    with pytest.raises(ValidationError) as e:
+        ci._validate_voxel_values(im)
+    assert e.value.message == (
+        "The valid voxel values for this segmentation are: {0, 1, 2}. "
+        "This segmentation is invalid as it contains the voxel values: {3}."
+    )
+    im = ImageFactoryWithImageFileTiff(segments=[1, 2])
+    assert ci._validate_voxel_values(im) is None
+
 
 @pytest.mark.django_db
 def test_can_execute():