Repository: comic/grand-challenge.org

Issue:
Mismatch in evaluation jobs when challenge admin pre-runs algorithm on cases from the phases archive
`create_algorithm_jobs_for_evaluation` exits successfully but the evaluation remains in the executing algorithm state. Occurs when the challenge admin uses the try out algorithm page and selects an image from the archive linked to the phase. Probably the jobs need filtering by creator?


Discussion:
I remember that we had this issue once before. I would have thought that we fixed it then... but I guess not. I will see if I can find our conversation back tomorrow.
Can't find anything... only that we used to filter by creator, but removed that here: https://github.com/comic/grand-challenge.org/commit/10fa528855b986914a76e0071b637c9780852a6e#diff-3de0492fca47670a7223a3218b626d8a5ae21bddebf88c6fc58499e64447eecd 
I think we just need to add that filter back.

I think this happened again for submission `e4d72823-3013-4c5a-a45f-7d848e107554` and evaluation `a883a7ff-052d-4593-b348-9cebb512d55e`. How do we fix this? Delete the algorithm job?
I think that would work.
I'll create an admin issue for it.
This happened again - is there a reason why we cannot add the filter back?
Not that I know. I don't know why it was removed in the first place. 
Me either - I removed it [in this commit](https://github.com/comic/grand-challenge.org/pull/2780/commits/42b640a720ad3a982746befa872d107a1cf125a0#diff-3de0492fca47670a7223a3218b626d8a5ae21bddebf88c6fc58499e64447eecd) which was removing batch job creation, unrelated to the evaluation. Think it must have been a mistake.
Let's put it back then. We will find out soon enough if there was a reason, I guess. 
I can create a PR

Resolving patch:
diff --git a/app/grandchallenge/algorithms/tasks.py b/app/grandchallenge/algorithms/tasks.py
--- a/app/grandchallenge/algorithms/tasks.py
+++ b/app/grandchallenge/algorithms/tasks.py
@@ -319,7 +319,7 @@ def filter_civs_for_algorithm(*, civ_sets, algorithm_image):
                 ),
             )
         )
-        .filter(inputs_match_count=len(input_interfaces))
+        .filter(inputs_match_count=len(input_interfaces), creator=None)
         .prefetch_related("inputs")
     }
 


Test patch:
diff --git a/app/tests/algorithms_tests/test_tasks.py b/app/tests/algorithms_tests/test_tasks.py
--- a/app/tests/algorithms_tests/test_tasks.py
+++ b/app/tests/algorithms_tests/test_tasks.py
@@ -639,13 +639,19 @@ def test_existing_jobs(self):
         cis = ComponentInterfaceFactory.create_batch(2)
         ai.algorithm.inputs.set(cis)
 
-        civs = [ComponentInterfaceValueFactory(interface=c) for c in cis]
+        civs1 = [ComponentInterfaceValueFactory(interface=c) for c in cis]
+        civs2 = [ComponentInterfaceValueFactory(interface=c) for c in cis]
 
-        j = AlgorithmJobFactory(algorithm_image=ai)
-        j.inputs.set(civs)
+        j1 = AlgorithmJobFactory(creator=None, algorithm_image=ai)
+        j1.inputs.set(civs1)
+        j2 = AlgorithmJobFactory(algorithm_image=ai)
+        j2.inputs.set(civs2)
 
         civ_sets = [
-            civs,  # Job already exists
+            {civ for civ in civs1},  # Job already exists (system job)
+            {
+                civ for civ in civs2
+            },  # Job already exists but with a creator set and hence should be ignored
             {
                 # New values
                 ComponentInterfaceValueFactory(interface=cis[0]),
@@ -653,7 +659,7 @@ def test_existing_jobs(self):
             },
             {
                 # Changed values
-                civs[0],
+                civs1[0],
                 ComponentInterfaceValueFactory(interface=cis[1]),
             },
         ]
@@ -662,7 +668,7 @@ def test_existing_jobs(self):
             civ_sets=civ_sets, algorithm_image=ai
         )
 
-        assert filtered_civ_sets == civ_sets[1:]
+        assert sorted(filtered_civ_sets) == sorted(civ_sets[1:])
 
 
 @pytest.mark.django_db