internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: comic/grand-challenge.org

Issue:
Handle NoneType comparison in _scores_to_ranks
```
TypeError: '<' not supported between instances of 'NoneType' and 'float'
```


Discussion:
Results json:

```
{
  "EADChallenge2019": {
    "IOU_d": {
      "value": null
    },
    "dev_g": {
      "value": 0.0
    },
    "mAP_d": {
      "value": 0.0
    },
    "mAP_g": {
      "value": 0.0
    },
    "overlap": {
      "value": 0
    },
    "score_d": {
      "value": null
    },
    "F2-score": {
      "value": 0
    },
    "semantic": {
      "value": 0
    }
  }
}
```

Resolving patch:
diff --git a/app/grandchallenge/evaluation/utils.py b/app/grandchallenge/evaluation/utils.py
--- a/app/grandchallenge/evaluation/utils.py
+++ b/app/grandchallenge/evaluation/utils.py
@@ -52,7 +52,10 @@ def _filter_valid_results(
     return [
         res
         for res in results
-        if all(get_jsonpath(res.metrics, m.path) != "" for m in metrics)
+        if all(
+            get_jsonpath(res.metrics, m.path) not in ["", None]
+            for m in metrics
+        )
     ]
 
 


Test patch:
diff --git a/app/tests/evaluation_tests/test_utils.py b/app/tests/evaluation_tests/test_utils.py
--- a/app/tests/evaluation_tests/test_utils.py
+++ b/app/tests/evaluation_tests/test_utils.py
@@ -231,6 +231,39 @@ def test_results_display(settings):
     assert_ranks(queryset, expected_ranks)
 
 
+@pytest.mark.django_db
+def test_null_results(settings):
+    # Override the celery settings
+    settings.task_eager_propagates = (True,)
+    settings.task_always_eager = (True,)
+    settings.broker_url = ("memory://",)
+    settings.backend = "memory"
+
+    challenge = ChallengeFactory()
+
+    with mute_signals(post_save):
+        user1 = UserFactory()
+        queryset = (
+            ResultFactory(
+                job__submission__challenge=challenge,
+                metrics={"a": 0.6},
+                job__submission__creator=user1,
+            ),
+            ResultFactory(
+                job__submission__challenge=challenge,
+                metrics={"a": None},
+                job__submission__creator=user1,
+            ),
+        )
+
+    challenge.evaluation_config.score_jsonpath = "a"
+    challenge.evaluation_config.result_display_choice = Config.ALL
+    challenge.evaluation_config.save()
+
+    expected_ranks = [1, 0]
+    assert_ranks(queryset, expected_ranks)
+
+
 def assert_ranks(queryset, expected_ranks, expected_rank_scores=None):
     for r in queryset:
         r.refresh_from_db()
Repository: comic/grand-challenge.org

Issue:
grandchallenge.cases.tasks.build_images should use a separate queue
This process can take a long time


Discussion:

Resolving patch:
diff --git a/app/config/settings.py b/app/config/settings.py
--- a/app/config/settings.py
+++ b/app/config/settings.py
@@ -601,7 +601,8 @@ def strtobool(val) -> bool:
 }
 
 CELERY_TASK_ROUTES = {
-    "grandchallenge.container_exec.tasks.execute_job": "evaluation"
+    "grandchallenge.container_exec.tasks.execute_job": "evaluation",
+    "grandchallenge.cases.tasks.build_images": "images",
 }
 
 # Set which template pack to use for forms


Test patch:
Repository: comic/grand-challenge.org

Issue:
Make the pid limit configurable
Sometimes we hit the pid limit, especially when running the workstations.


Discussion:

Resolving patch:
diff --git a/app/config/settings.py b/app/config/settings.py
--- a/app/config/settings.py
+++ b/app/config/settings.py
@@ -582,6 +582,12 @@ def strtobool(val) -> bool:
 CONTAINER_EXEC_CPU_PERIOD = int(
     os.environ.get("CONTAINER_EXEC_CPU_PERIOD", "100000")
 )
+CONTAINER_EXEC_PIDS_LIMIT = int(
+    os.environ.get("CONTAINER_EXEC_PIDS_LIMIT", "128")
+)
+CONTAINER_EXEC_CPU_SHARES = int(
+    os.environ.get("CONTAINER_EXEC_CPU_SHARES", "1024")  # Default weight
+)
 CONTAINER_EXEC_DOCKER_RUNTIME = os.environ.get(
     "CONTAINER_EXEC_DOCKER_RUNTIME", None
 )
diff --git a/app/grandchallenge/container_exec/backends/docker.py b/app/grandchallenge/container_exec/backends/docker.py
--- a/app/grandchallenge/container_exec/backends/docker.py
+++ b/app/grandchallenge/container_exec/backends/docker.py
@@ -64,12 +64,11 @@ def __init__(
             "memswap_limit": settings.CONTAINER_EXEC_MEMORY_LIMIT,
             "cpu_period": settings.CONTAINER_EXEC_CPU_PERIOD,
             "cpu_quota": settings.CONTAINER_EXEC_CPU_QUOTA,
-            # Use the default weight
-            "cpu_shares": 1024,
+            "cpu_shares": settings.CONTAINER_EXEC_CPU_SHARES,
             "runtime": settings.CONTAINER_EXEC_DOCKER_RUNTIME,
             "cap_drop": ["all"],
             "security_opt": ["no-new-privileges"],
-            "pids_limit": 128,
+            "pids_limit": settings.CONTAINER_EXEC_PIDS_LIMIT,
             "log_config": LogConfig(
                 type=LogConfig.types.JSON, config={"max-size": "1g"}
             ),


Test patch:
Repository: comic/grand-challenge.org

Issue:
Update to Django 2.2
- [ ] Check that all Django apps support 2.2


Discussion:

Resolving patch:
diff --git a/app/grandchallenge/serving/views.py b/app/grandchallenge/serving/views.py
--- a/app/grandchallenge/serving/views.py
+++ b/app/grandchallenge/serving/views.py
@@ -101,7 +101,7 @@ def serve_folder(request, *, challenge_name=None, folder=None, path):
 
 
 def serve_images(request, *, pk, path):
-    document_root = safe_join(f"/{settings.IMAGE_FILES_SUBDIRECTORY}", pk)
+    document_root = safe_join(f"/{settings.IMAGE_FILES_SUBDIRECTORY}", str(pk))
     path = posixpath.normpath(path).lstrip("/")
     name = safe_join(document_root, path)
 


Test patch:
Repository: comic/grand-challenge.org

Issue:
Remove unused CIRRUS environment variables
These variables are now deprecated.

https://github.com/comic/grand-challenge.org/blob/30875fd388f2ad14212cf57c0caa8b9efcba19d9/app/grandchallenge/workstations/models.py#L201


Discussion:

Resolving patch:
diff --git a/app/grandchallenge/workstations/models.py b/app/grandchallenge/workstations/models.py
--- a/app/grandchallenge/workstations/models.py
+++ b/app/grandchallenge/workstations/models.py
@@ -197,12 +197,7 @@ def environment(self) -> dict:
         -------
             The environment variables that should be set on the container.
         """
-        env = {
-            "GRAND_CHALLENGE_PROXY_URL_MAPPINGS": "",
-            "GRAND_CHALLENGE_QUERY_IMAGE_URL": unquote(
-                reverse("api:image-detail", kwargs={"pk": "{key}"})
-            ),
-        }
+        env = {"GRAND_CHALLENGE_API_ROOT": unquote(reverse("api:api-root"))}
 
         if self.creator:
             env.update(


Test patch:
diff --git a/app/tests/workstations_tests/test_models.py b/app/tests/workstations_tests/test_models.py
--- a/app/tests/workstations_tests/test_models.py
+++ b/app/tests/workstations_tests/test_models.py
@@ -23,11 +23,7 @@ def test_session_environ(settings, debug):
     s = SessionFactory()
     env = s.environment
 
-    assert env["GRAND_CHALLENGE_PROXY_URL_MAPPINGS"] == ""
-    assert "{key}" in env["GRAND_CHALLENGE_QUERY_IMAGE_URL"]
-    assert env["GRAND_CHALLENGE_QUERY_IMAGE_URL"].startswith(
-        "https://testserver"
-    )
+    assert env["GRAND_CHALLENGE_API_ROOT"] == "https://testserver/api/v1/"
     assert (
         Token.objects.get(user=s.creator).key
         in env["GRAND_CHALLENGE_AUTHORIZATION"]
Repository: comic/grand-challenge.org

Issue:
Retina API tests to fail with DRF 3.10
See https://travis-ci.org/comic/grand-challenge.org/builds/562447770

Maybe from Django Rest Framework going from 3.9 to 3.10? https://github.com/comic/grand-challenge.org/compare/update_deps?expand=1#diff-46bb8405eeb4d8a11deb635f562f7c48R297

TODO:

- [ ] Fix tests
- [ ] Unpin `djangorestframework` in `Pipfile`
- [ ] `pipenv lock`


Discussion:
> Maybe from Django Rest Framework going from 3.9 to 3.10? https://github.com/comic/grand-challenge.org/compare/update_deps?expand=1#diff-46bb8405eeb4d8a11deb635f562f7c48R297

Confirmed with https://travis-ci.org/comic/grand-challenge.org/builds/562453979

Resolving patch:
diff --git a/app/grandchallenge/annotations/migrations/0006_auto_20190730_1234.py b/app/grandchallenge/annotations/migrations/0006_auto_20190730_1234.py
new file mode 100644
--- /dev/null
+++ b/app/grandchallenge/annotations/migrations/0006_auto_20190730_1234.py
@@ -0,0 +1,22 @@
+# Generated by Django 2.2.3 on 2019-07-30 12:34
+
+import django.contrib.postgres.fields
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [("annotations", "0005_imagetextannotation")]
+
+    operations = [
+        migrations.AlterField(
+            model_name="etdrsgridannotation",
+            name="optic_disk",
+            field=django.contrib.postgres.fields.ArrayField(
+                base_field=models.FloatField(),
+                blank=True,
+                default=list,
+                size=2,
+            ),
+        )
+    ]
diff --git a/app/grandchallenge/annotations/models.py b/app/grandchallenge/annotations/models.py
--- a/app/grandchallenge/annotations/models.py
+++ b/app/grandchallenge/annotations/models.py
@@ -240,7 +240,9 @@ class ETDRSGridAnnotation(AbstractImageAnnotationModel):
 
     # Fields for location of fovea and optic disk on the images: (x,y) coordinates
     fovea = ArrayField(models.FloatField(), size=2)
-    optic_disk = ArrayField(models.FloatField(), size=2)
+    optic_disk = ArrayField(
+        models.FloatField(), size=2, default=list, blank=True
+    )
 
     class Meta(AbstractImageAnnotationModel.Meta):
         unique_together = ("image", "grader", "created")


Test patch:
Repository: comic/grand-challenge.org

Issue:
Fix read the docs generation
We're using autodoc which requires an install of the dev environment. Read the docs does not support pipenv yet, so we should be able to fix when this PR is released: https://github.com/rtfd/readthedocs.org/pull/4783

For the time being, local generation works fine. 


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -45,6 +45,7 @@
     "sphinx.ext.viewcode",
     "sphinx.ext.napoleon",
     "sphinx_autodoc_typehints",
+    "sphinx_rtd_theme",
 ]
 
 napoleon_google_docstring = False
@@ -101,7 +102,7 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = "alabaster"
+html_theme = "sphinx_rtd_theme"
 
 # Theme options are theme-specific and customize the look and feel of a theme
 # further.  For a list of options available for each theme, see the


Test patch:
Repository: comic/grand-challenge.org

Issue:
Let users know that a result is under review on the evaluation jobs list page
From an email with a participant

**Is your feature request related to a problem? Please describe.**
>  After submitting the results, it says "Succeeded" yet no output (the evaluation results of my results) return. This happens several times, sometime I should wait one or two days, then I can access to the results, yet sometimes it just never returns.

![image (1)](https://user-images.githubusercontent.com/12661555/66252056-2b87c580-e757-11e9-9f62-813557fb2768.png)

**Describe the solution you'd like**
> Yes, for this challenge the organisers like to review each submission before it is released to the participant, I will add more information to that page (putting "Awaiting organiser review" for that result or something).


Discussion:

Resolving patch:
diff --git a/app/grandchallenge/core/templatetags/pathlib.py b/app/grandchallenge/core/templatetags/pathlib.py
new file mode 100644
--- /dev/null
+++ b/app/grandchallenge/core/templatetags/pathlib.py
@@ -0,0 +1,12 @@
+from pathlib import Path
+
+from django import template
+from django.template.defaultfilters import stringfilter
+
+register = template.Library()
+
+
+@register.filter
+@stringfilter
+def suffix(value):
+    return Path(value).suffix


Test patch:
Repository: comic/grand-challenge.org

Issue:
Replace pipenv with poetry
Pipenv development seems to have stalled and there are several bugs that I'm not convinced will be fixed anytime soon, we should migrate to poetry.


Discussion:

Resolving patch:
diff --git a/app/config/settings.py b/app/config/settings.py
--- a/app/config/settings.py
+++ b/app/config/settings.py
@@ -184,6 +184,7 @@ def strtobool(val) -> bool:
         "LOCATION": "memcached:11211",
     }
 }
+SPEEDINFO_STORAGE = "speedinfo.storage.cache.storage.CacheStorage"
 
 ROOT_URLCONF = "config.urls"
 SUBDOMAIN_URL_CONF = "grandchallenge.subdomains.urls"


Test patch:
diff --git a/app/tests/core_tests/test_formatting.py b/app/tests/core_tests/test_formatting.py
--- a/app/tests/core_tests/test_formatting.py
+++ b/app/tests/core_tests/test_formatting.py
@@ -2,5 +2,7 @@
 
 
 def test_code_is_black():
-    res = call(["black", "--check", "--config", "/tmp/pyproject.toml", "/app"])
+    res = call(
+        ["black", "--check", "--config", "/opt/poetry/pyproject.toml", "/app"]
+    )
     assert res == 0
Repository: comic/grand-challenge.org

Issue:
Fix spelling error for NearestNeighbour
**Describe the bug**
A clear and concise description of what the bug is.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

**Expected behavior**
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Smartphone (please complete the following information):**
 - Device: [e.g. iPhone6]
 - OS: [e.g. iOS8.1]
 - Browser [e.g. stock browser, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.



Discussion:

Resolving patch:
diff --git a/app/grandchallenge/workstation_configs/migrations/0002_auto_20191025_0743.py b/app/grandchallenge/workstation_configs/migrations/0002_auto_20191025_0743.py
new file mode 100644
--- /dev/null
+++ b/app/grandchallenge/workstation_configs/migrations/0002_auto_20191025_0743.py
@@ -0,0 +1,21 @@
+# Generated by Django 2.2.6 on 2019-10-25 07:43
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [("workstation_configs", "0001_initial")]
+
+    operations = [
+        migrations.AlterField(
+            model_name="workstationconfig",
+            name="default_overlay_interpolation",
+            field=models.CharField(
+                blank=True,
+                choices=[("NN", "NearestNeighbor"), ("TL", "Trilinear")],
+                default="NN",
+                max_length=2,
+            ),
+        )
+    ]
diff --git a/app/grandchallenge/workstation_configs/models.py b/app/grandchallenge/workstation_configs/models.py
--- a/app/grandchallenge/workstation_configs/models.py
+++ b/app/grandchallenge/workstation_configs/models.py
@@ -31,7 +31,7 @@ class WorkstationConfig(TitleSlugDescriptionModel, UUIDModel):
     IMAGE_INTERPOLATION_TYPE_NEAREST = "NN"
     IMAGE_INTERPOLATION_TYPE_TRILINEAR = "TL"
     IMAGE_INTERPOLATION_TYPE_CHOICES = (
-        (IMAGE_INTERPOLATION_TYPE_NEAREST, "NearestNeighbour"),
+        (IMAGE_INTERPOLATION_TYPE_NEAREST, "NearestNeighbor"),
         (IMAGE_INTERPOLATION_TYPE_TRILINEAR, "Trilinear"),
     )
 


Test patch:
Repository: comic/grand-challenge.org

Issue:
URLs cannot be accessed without trailing slash
**Describe the bug**

URLs within grand-challenge result in a 404 when the trailing slash is missing.

**To Reproduce**
Steps to reproduce the behavior:
1. Go to https://grand-challenge.org/algorithms
2. Results in 404

**Expected behavior**

https://grand-challenge.org/algorithms/ works. I expect to see that page (or to be redirected to the page with the trailing slash. 

**Additional context**

I expect, that when sharing the link tot the algorithms page, often people will forget the trailing slash. Would be nice if the URL routing can handle both scenarios (with a canonical url set). Could also be done on the server by setting a 301 to the URL with trailing slash.



Discussion:

Resolving patch:
diff --git a/app/config/settings.py b/app/config/settings.py
--- a/app/config/settings.py
+++ b/app/config/settings.py
@@ -282,6 +282,8 @@ def strtobool(val) -> bool:
     "grandchallenge.subdomains.middleware.subdomain_middleware",
     "grandchallenge.subdomains.middleware.challenge_subdomain_middleware",
     "grandchallenge.subdomains.middleware.subdomain_urlconf_middleware",
+    # Flatpage fallback almost last
+    "django.contrib.flatpages.middleware.FlatpageFallbackMiddleware",
     # speedinfo at the end but before FetchFromCacheMiddleware
     "speedinfo.middleware.ProfilerMiddleware",
 )
diff --git a/app/config/urls.py b/app/config/urls.py
--- a/app/config/urls.py
+++ b/app/config/urls.py
@@ -98,11 +98,6 @@ def handler500(request):
         "media/",
         include("grandchallenge.serving.urls", namespace="root-serving"),
     ),
-    # ========== catch all ====================
-    # when all other urls have been checked, try to load page from flatpages
-    # keep this url at the bottom of this list, because urls are checked in
-    # order
-    path("", include("django.contrib.flatpages.urls")),
 ]
 if settings.DEBUG and settings.ENABLE_DEBUG_TOOLBAR:
     import debug_toolbar
diff --git a/app/grandchallenge/subdomains/utils.py b/app/grandchallenge/subdomains/utils.py
--- a/app/grandchallenge/subdomains/utils.py
+++ b/app/grandchallenge/subdomains/utils.py
@@ -24,11 +24,6 @@ def reverse(viewname, urlconf=None, args=None, kwargs=None, current_app=None):
     else:
         urlconf = urlconf or settings.ROOT_URLCONF
 
-    if "views.flatpage" in viewname.lower() and "url" in kwargs:
-        # Fix for a long standing bug in django flatpages
-        # https://code.djangoproject.com/ticket/15658
-        kwargs.update({"url": kwargs["url"].lstrip("/")})
-
     path = reverse_org(
         viewname,
         urlconf=urlconf,


Test patch:
Repository: comic/grand-challenge.org

Issue:
As a reader study editor would like to be able to see why a reader study can't be used (yet).
**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.



Discussion:
A student had some issues with this so bumping the priority, it's only a change to the templates so a small fix.

Resolving patch:
diff --git a/app/grandchallenge/reader_studies/models.py b/app/grandchallenge/reader_studies/models.py
--- a/app/grandchallenge/reader_studies/models.py
+++ b/app/grandchallenge/reader_studies/models.py
@@ -186,7 +186,7 @@ def remove_reader(self, user):
 
     @property
     def study_image_names(self):
-        return [im.name for im in self.images.all()]
+        return self.images.values_list("name", flat=True)
 
     @property
     def hanging_image_names(self):
@@ -204,6 +204,15 @@ def hanging_list_valid(self):
             self.hanging_image_names
         )
 
+    @property
+    def hanging_list_diff(self):
+        return {
+            "in_study_list": set(self.study_image_names)
+            - set(self.hanging_image_names),
+            "in_hanging_list": set(self.hanging_image_names)
+            - set(self.study_image_names),
+        }
+
     @property
     def non_unique_study_image_names(self):
         """


Test patch:
Repository: comic/grand-challenge.org

Issue:
"Run algorithm" page is missing the breadcrumb bar
Maybe this is on purpose, but I find it confusing that the page that allows users to upload images and run an algorithm does not show the breadcrumb bar to get back to the algorithm overview page.

Example: https://grand-challenge.org/algorithms/vertebra-segmentation/run/

To be consistent with the job list the title of that page could also read "Run this algorithm" instead "Run an algorithm"


Discussion:
Related: #994 - maybe review all breadcrumbs for these apps.
Also missing on the "add user" page: https://grand-challenge.org/algorithms/vertebra-segmentation/users/update/

Resolving patch:
diff --git a/app/grandchallenge/reader_studies/views.py b/app/grandchallenge/reader_studies/views.py
--- a/app/grandchallenge/reader_studies/views.py
+++ b/app/grandchallenge/reader_studies/views.py
@@ -157,7 +157,7 @@ def get_context_data(self, **kwargs):
         for field_name in self.object.read_only_fields:
             form_fields[field_name].required = False
             form_fields[field_name].disabled = True
-
+        context.update({"reader_study": self.reader_study})
         return context
 
     def get_success_url(self):


Test patch:
Repository: comic/grand-challenge.org

Issue:
Ability to remove readers, editors (and users)
We need to be able to remove Readers from a reader study as an Editor. Could this be added?


Discussion:
That should not be a problem. Do you think we should also give editors the option to remove other editors or should we leave that as a task for admins?
> That should not be a problem. Do you think we should also give editors the option to remove other editors or should we leave that as a task for admins?

We let challenge admins do this, so yes, I would say this is fine.

Resolving patch:
diff --git a/app/grandchallenge/reader_studies/views.py b/app/grandchallenge/reader_studies/views.py
--- a/app/grandchallenge/reader_studies/views.py
+++ b/app/grandchallenge/reader_studies/views.py
@@ -113,8 +113,19 @@ def get_context_data(self, **kwargs):
                 self.request.user
             )
             context.update({"progress": user_progress})
+
+        reader_remove_form = ReadersForm()
+        reader_remove_form.fields["action"].initial = ReadersForm.REMOVE
+        editor_remove_form = EditorsForm()
+        editor_remove_form.fields["action"].initial = EditorsForm.REMOVE
         context.update(
-            {"user_is_reader": self.object.is_reader(user=self.request.user)}
+            {
+                "editor_remove_form": editor_remove_form,
+                "reader_remove_form": reader_remove_form,
+                "user_is_reader": self.object.is_reader(
+                    user=self.request.user
+                ),
+            }
         )
         return context
 


Test patch: