Repository: comic/grand-challenge.org

Issue:
Adding of duplicate CIs to an archive item
The set of CIs for an archive item should be unique. However, it is currently possible to add multiple CIV with the same CI for a single archive item. This is because the check to see if an a CI already exists and should be removed for an archive item is done asynchronously. So while the task to add a CIV to an item is queued, another task can be spawned to add another CIV for the same CI. This will result in both CIVs being added to the item so duplicate CIs on a single item. 

I'm not sure how to mitigate this issue. We could try to check the task queue (if that is even possible) before adding a new task to add a CIV if there are already tasks for that CI in it. Then the new task should be cancelled/not triggered. Because of the asynchronous nature it will then probably be impossible to properly show an error about this to the user. Probably there is a better way, but I'm not seeing it currently.

This issue is further magnified because it is not clear that an upload is processing when adding a new CIV to an archive item. When you add a file type CIV to an archive item it will redirect you back to the archive item list view. When the CIV adding task is still queued after you are redirected which is quite likely, you will not see any indication that it is in the process of being added on that page. Only when you check https://grand-challenge.org/cases/uploads/ you can see the upload session status. So a user could think something went wrong and try to add it again which will result in the issue described above. Maybe it is good to redirect the user to the cases/uploads page after adding a CIV so that they see that the task is queued.


Discussion:
What I think we should do:

Rather than calling `update_archive_item_update_kwargs` separetly in the [view](https://github.com/comic/grand-challenge.org/blob/main/app/grandchallenge/archives/views.py#L419) and in the [serializer](https://github.com/comic/grand-challenge.org/blob/main/app/grandchallenge/archives/serializers.py#L99), it should be chained in the `start_archive_item_update_tasks` task. I think the issue now arises because the the updating is separate from the initial check, which means they can run in parallel, while there is a dependency and the check should always happen before the actual update. What do you think @jmsmkn ?
Don't think that is it, although `update_archive_item_update_kwargs` is in `tasks.py` it is not a Celery task, and is being called synchronously, so will always be run before `start_archive_item_update_tasks` anyway.
Ah I didn't look properly, sorry about that.
However, wouldn't making it a task and chaining it in front of the other tasks guarantee that there is no time between the check and the execution of the rest of the code? Or alternatively, we can make the check part of the update task. Now the check gets performed on form save, while the rest of the logic gets added to the queue and might not execute until a lot later.
Chaining just creates a list of tasks, with each task not being queued until the previous one was completed. It does not guarantee time between two tasks, each new task always gets put on the back of the queue so it can still take a while. If you have concerns about timing then everything needs to be done in the same transaction.
I wasn't sure on that I I couldn't find a definitive answer in the docs either. In that case I'd suggest just adding the checks to the update task. We should probably move it to the 2xl queue as well.

Resolving patch:
diff --git a/app/grandchallenge/archives/serializers.py b/app/grandchallenge/archives/serializers.py
--- a/app/grandchallenge/archives/serializers.py
+++ b/app/grandchallenge/archives/serializers.py
@@ -85,7 +85,6 @@ def __init__(self, *args, **kwargs):
     def update(self, instance, validated_data):
         civs = validated_data.pop("values")
 
-        civ_pks_to_remove = set()
         civ_pks_to_add = set()
         upload_pks = {}
 
@@ -104,7 +103,6 @@ def update(self, instance, validated_data):
                 user_upload=user_upload,
                 upload_session=upload_session,
                 civ_pks_to_add=civ_pks_to_add,
-                civ_pks_to_remove=civ_pks_to_remove,
                 upload_pks=upload_pks,
             )
 
@@ -113,7 +111,6 @@ def update(self, instance, validated_data):
                 kwargs={
                     "archive_item_pk": instance.pk,
                     "civ_pks_to_add": list(civ_pks_to_add),
-                    "civ_pks_to_remove": list(civ_pks_to_remove),
                     "upload_pks": upload_pks,
                 }
             ).apply_async
diff --git a/app/grandchallenge/archives/tasks.py b/app/grandchallenge/archives/tasks.py
--- a/app/grandchallenge/archives/tasks.py
+++ b/app/grandchallenge/archives/tasks.py
@@ -74,21 +74,10 @@ def add_images_to_archive_item(
         )
 
 
-@shared_task(**settings.CELERY_TASK_DECORATOR_KWARGS["acks-late-micro-short"])
-def update_archive_item_values(
-    *, archive_item_pk, civ_pks_to_remove, civ_pks_to_add
-):
-    with transaction.atomic():
-        archive_item = ArchiveItem.objects.get(pk=archive_item_pk)
-        archive_item.values.remove(*civ_pks_to_remove)
-        archive_item.values.add(*civ_pks_to_add)
-
-
 def update_archive_item_update_kwargs(
     instance,
     interface,
     civ_pks_to_add,
-    civ_pks_to_remove,
     upload_pks,
     value=None,
     image=None,
@@ -104,26 +93,16 @@ def update_archive_item_update_kwargs(
     it also appends the session pk together with the new civ pk to the list of
     to be processed images.
     """
-    if instance.values.filter(interface=interface.pk).exists():
-        for civ_pk in instance.values.filter(
-            interface=interface.pk
-        ).values_list("pk", flat=True):
-            civ_pks_to_remove.add(civ_pk)
-    else:
-        # for images, check if there are any CIVs with the provided image
-        if interface.is_image_kind:
-            if instance.values.filter(image=image).exists():
-                for civ_pk in instance.values.filter(image=image).values_list(
-                    "pk", flat=True
-                ):
-                    civ_pks_to_remove.add(civ_pk)
-
     with transaction.atomic():
         if interface.is_image_kind:
-            civ = ComponentInterfaceValue.objects.create(interface=interface)
             if image:
-                civ.image = image
-                civ.full_clean()
+                civ = ComponentInterfaceValue.objects.filter(
+                    interface=interface, image=image
+                ).first()
+                if civ is None:
+                    civ = ComponentInterfaceValue.objects.create(
+                        interface=interface, image=image
+                    )
             elif upload_session:
                 upload_pks[civ.pk] = upload_session.pk
             civ.save()
@@ -139,18 +118,39 @@ def update_archive_item_update_kwargs(
             civ = interface.create_instance(value=value)
             civ_pks_to_add.add(civ.pk)
 
-    return civ_pks_to_add, civ_pks_to_remove, upload_pks
+
+@shared_task(**settings.CELERY_TASK_DECORATOR_KWARGS["acks-late-micro-short"])
+def update_archive_item_values(*, archive_item_pk, civ_pks_to_add):
+    instance = ArchiveItem.objects.get(pk=archive_item_pk)
+    civ_pks_to_remove = []
+    civs = ComponentInterfaceValue.objects.filter(pk__in=civ_pks_to_add)
+    for civ in civs:
+        if instance.values.filter(interface=civ.interface.pk).exists():
+            for civ_pk in instance.values.filter(
+                interface=civ.interface.pk
+            ).values_list("pk", flat=True):
+                civ_pks_to_remove.append(civ_pk)
+        # for images, check if there are any CIVs with the provided image
+        if civ.interface.is_image_kind:
+            if instance.values.filter(image=civ.image).exists():
+                for civ_pk in instance.values.filter(
+                    image=civ.image
+                ).values_list("pk", flat=True):
+                    civ_pks_to_remove.append(civ_pk)
+
+    with transaction.atomic():
+        instance.values.remove(*civ_pks_to_remove)
+        instance.values.add(*civ_pks_to_add)
 
 
 @shared_task(**settings.CELERY_TASK_DECORATOR_KWARGS["acks-late-micro-short"])
 def start_archive_item_update_tasks(
-    archive_item_pk, civ_pks_to_add, civ_pks_to_remove, upload_pks
+    archive_item_pk, civ_pks_to_add, upload_pks
 ):
     tasks = update_archive_item_values.signature(
         kwargs={
             "archive_item_pk": archive_item_pk,
             "civ_pks_to_add": civ_pks_to_add,
-            "civ_pks_to_remove": civ_pks_to_remove,
         },
         immutable=True,
     )
diff --git a/app/grandchallenge/archives/views.py b/app/grandchallenge/archives/views.py
--- a/app/grandchallenge/archives/views.py
+++ b/app/grandchallenge/archives/views.py
@@ -1,3 +1,4 @@
+from django.contrib import messages
 from django.contrib.auth.mixins import PermissionRequiredMixin
 from django.contrib.messages.views import SuccessMessageMixin
 from django.core.exceptions import (
@@ -394,6 +395,8 @@ def get_context_data(self, *args, **kwargs):
 
     def form_valid(self, form):
         def create_upload(image_files):
+            if not image_files:
+                return
             upload_session = RawImageUploadSession.objects.create(
                 creator=self.request.user
             )
@@ -401,7 +404,6 @@ def create_upload(image_files):
             return upload_session
 
         upload_pks = {}
-        civ_pks_to_remove = set()
         civ_pks_to_add = set()
 
         for slug, value in form.cleaned_data.items():
@@ -411,8 +413,7 @@ def create_upload(image_files):
             ci = ComponentInterface.objects.get(slug=slug)
 
             if ci.is_image_kind:
-                if value:
-                    upload_session = create_upload(value)
+                upload_session = create_upload(value)
             else:
                 upload_session = None
 
@@ -423,7 +424,6 @@ def create_upload(image_files):
                 user_upload=value if ci.is_file_kind else None,
                 upload_session=upload_session,
                 civ_pks_to_add=civ_pks_to_add,
-                civ_pks_to_remove=civ_pks_to_remove,
                 upload_pks=upload_pks,
             )
 
@@ -432,12 +432,16 @@ def create_upload(image_files):
                 kwargs={
                     "archive_item_pk": self.archive_item.pk,
                     "civ_pks_to_add": list(civ_pks_to_add),
-                    "civ_pks_to_remove": list(civ_pks_to_remove),
                     "upload_pks": upload_pks,
                 }
             ).apply_async
         )
-
+        messages.add_message(
+            self.request,
+            messages.SUCCESS,
+            "Archive item will be updated. It may take some time for your "
+            "changes to become visible.",
+        )
         return HttpResponseRedirect(
             reverse("archives:items-list", kwargs={"slug": self.archive.slug})
         )


Test patch: