Repository: ome/omero-web (language: python)

Issue:
WIP:Django upgrade to 2.2
--exclude

This is still work in progress, opening a PR to collect feedback and push changes accordingly.
Doing it serially to start with from the current version to 2.0 and then potentially to 2.2

Trying to not touch the regex's as of now, wanted to the transition to using converter specifications in the url pattern but have avoided it for the time being. 

Happy to take feedback and contribute accordingly.
@will-moore @joshmoore 


Discussion / hints:
Hi Balaji,
Great that you're able to help with this!
Have you looked at the work I started on this a while back? https://github.com/will-moore/omero-web/tree/django_22

I haven't worked on it for a little while, so the branch is out of date and needs origin/master merging back in or rebasing.
But it looks like you're going through the same steps as me, so it might save you some work.

I certainly prefer the new `path()` syntax compared with using regexes, although I found we still need regexes for some cases.

 Cheers!
  Will.
 
Hi Will,

I saw the PR, thank you 
I am just following the official Django documentation and release notes and running tests before every commit,
https://docs.djangoproject.com/en/3.1/howto/upgrade-version/

thus following the same path as you (to do sanity checks after every step). I will eventually be moving all your commits as well.
This process just helped me to get to the bottom of the release notes (and understand the choices you have made as well).

I started with using `path` as well,
https://github.com/bramalingam/omero-web/tree/django_upgrade
but it meant touching most of the regex and it looked like a lot of breaking changes and I wasn't sure about our testing coverage as it stands (for all url-paths).

If the target is to move towards a stable version of 2.2 quickly (re_path usage looked like the least breaking path). 

Happy to revert back to `path` because it improves readability of the URL patterns and provides type safety as well with `<int:objectid>`. Will try and push commits for one module at a time in that case (to make reviewing easier), if you prefer this route. Please let me know either way.
@will-moore : I have replaced the usage of re_path with path in the most obvious cases.
All urls with optional arguments still use re_path (Happy to take some feedback on this front). 
This currently improves readability a bit. 

Will push the following this week,
1) Cherry-pick the rest of your commits from your PR.
2) Upgrade to using version 2.2
3) Fix Flake8 issues.
4) Integrate with 'django_extensions'

I am also going to look into writing some unit tests to test the minimum,
`assertEqual(200, response.status_code)`
for all the urls that I have touched (if they are missing). 
Happy to take some feedback on this front, as well.



Hi Balaji,
 Almost all the view methods are wrapped with the `@login_required` decorator which will return a 403 (not logged in) unless you are connected to OMERO. So for unit tests I think you'll be restricted to testing for that. That won't tell you much, but it will at least tell you it's not a 404.
@will-moore : I have cherry-picked your commits into this branch and started testing the URL's manually to start with.

I identified a primary issue with 2.2 upgrade being the following,
https://docs.djangoproject.com/en/3.1/releases/2.1/
specifically: url()’s support for inline flags in regular expression groups ((?i), (?L), (?m), (?s), and (?u)) is removed

and I found this following PR,
https://github.com/ome/omero-web/pull/93

Looks like you had started some work on this front as well. Will try and resolve this today.

@joshmoore : Meanwhile, it looks like black formatting PR has gone in,
Any suggestions on a nice way to rebase this PR? 
Hopefully, rebasing should sort out all the Flake8 issues in this PR
@will-moore : Reg case insensitive urls, here are the 3 options that I could find,

1) Dropping support for case insensitive urls (am guessing this is an extreme and you wouldn't want this).
2) Registering a custom path converter
https://docs.djangoproject.com/en/2.2/topics/http/urls/#registering-custom-path-converters
This looks like the most viable option according to me, happy to take your feedback on the same.
3) Using a middleware (given its performance implications, this might not be a great option as well).

Thought I'd take some guidance before proceeding with the implementation.


> Any suggestions on a nice way to rebase this PR?

If you can either squash all the commits then run pre-commit on the sum or you can try to `rebase -i` running pre-commit on each commit.
I _think_ running pre-commit on your branch, then merging in `master` and resolving conflicts, should also work.
Thank you @joshmoore and @manics : will try and update
I tried something last night - see https://github.com/will-moore/omero-web/tree/django_upgrade_re_path
That tries to merge everything, with black. But I haven't tried running it yet!
Hi Will, given the issue with (?i) being there in our urls to generate case insensitive urls the current branch is still not ready to run. I will try and push a change with the converters and test your branch as well, on first look it looks perfect (thanks for fixing the default value for handler404 as well).
Just starting to test locally, I realise that all the `omero.web.apps` also need to work with Django 2.2.
Principle ones are `omero-figure` and `omero-iviewer`. I'm going to remove all these apps initially for testing, but then we need to look at upgrading the various apps.
For my own records, I currently have these locally:
```
$ omero config get omero.web.apps ["omero_iviewer", "parade_crossfilter", "omero_figure", "corsheaders", "omero_parade", "omero_webtest", "my_channel_viewer", "omero_mapr", "omero_openwith_url"]
```

@bramalingam Travis is failing:

```
black....................................................................Failed
- hook id: black
- files were modified by this hook
reformatted /home/travis/build/ome/omero-web/omeroweb/settings.py
```

So I think that file needs `black` formatting applied.
Testing locally, a quick browse around the webclient looks good. The major blocker on having this included in our daily merge-ci build is the `omero.web.apps` mentioned above.
The current list there is:
 - [ ] omero_figure
 - [ ] omero_iviewer
 - [ ] omero_parade
 - [ ] omero_fpbioimage
 - [ ] omero_gallery
 - [ ] omero_mapr
 - [ ] omero_weberror
 - [ ] omero_webtest
 - [ ] omero_webtagging_autotag
 - [ ] omero_webtagging_tagsearch
 - [ ] omero_openwith_url
 - [ ] parade_crossfilter (github.com/will-moore/parade-crossfilter)

We don't need to have all of these updated before we start including this PR in the merge build (we can temporarily comment-out the install of some) but at least figure & iviewer. Then follow up with mapr and gallery ASAP, then fpbioimage and the rest...

As with the last Django upgrade, this shouldn't take very long to tackle each app.
Thank you for the detailed review @will-moore. I will fix the black formatting and push a commit. Happy to create separate PR's for iViewer and figure as well once this is merged.  I still haven't finished the unit-tests, will wrap up whatever I have this weekend and push to this branch.
@will-moore : I've added a test that tests URLs which do not use any arguments, and covers a fraction of URLs. And I've fixed the black formatting as well. The test could be further improved to make sure URLs with arguments are handled as well. The test is just to make sure that I have not broken any URL patterns. I am happy to get started with the PR's for Figure & iviewer, if that sounds good to you. 
@bramalingam Sounds great. Figure and iviewer don't have so much Python code and not many URLs, so hopefully won't be much to fix. Thanks.
Hi @will-moore : I have opened up two PR's for figure and viewer, it would be great if they could be integrated with this PR and tested (locally) by someone. Am happy to open PR's for the other apps if all these changes look sensible and are not breaking any existing urls. 
Trying to run this again... New python3.9 environment:
```
$ conda create -n omeroweb2 -c ome python=3.9 zeroc-ice36-python omero-py
$ conda activate omeroweb2
# install omero-web from this branch...
$ pip install -e .
```

```
$ python omeroweb/manage.py runserver 4080
...
# Trying to load login page...
# few minutes delay, then...

Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/wsgiref/handlers.py", line 138, in run
    self.finish_response()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/wsgiref/handlers.py", line 184, in finish_response
    self.write(data)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/wsgiref/handlers.py", line 288, in write
    self.send_headers()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/wsgiref/handlers.py", line 347, in send_headers
    self._write(bytes(self.headers))
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/wsgiref/handlers.py", line 467, in _write
    result = self.stdout.write(data)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 826, in write
    self._sock.sendall(b)
OSError: [Errno 41] Protocol wrong type for socket
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52363)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 171, in handle
    self.handle_one_request()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52370)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 171, in handle
    self.handle_one_request()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52372)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 171, in handle
    self.handle_one_request()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52374)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 171, in handle
    self.handle_one_request()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
----------------------------------------
Exception occurred during processing of request from ('127.0.0.1', 52376)
Traceback (most recent call last):
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 316, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 347, in process_request
    self.finish_request(request, client_address)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 360, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socketserver.py", line 747, in __init__
    self.handle()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 171, in handle
    self.handle_one_request()
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/site-packages/django/core/servers/basehttp.py", line 179, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/opt/anaconda3/envs/omeroweb2/lib/python3.9/socket.py", line 704, in readinto
    return self._sock.recv_into(b)
ConnectionResetError: [Errno 54] Connection reset by peer
----------------------------------------
2022-02-08 16:46:14,179  INFO [                           django.server] (proc.47304) log_message():154 "GET /static/webclient/css/dusty.css?_5.8.1.dev0 HTTP/1.1" 200 36841
```

Fix patch:
diff --git a/omero/plugins/web.py b/omero/plugins/web.py
index 078f76b613..ac1b7fa312 100644
--- a/omero/plugins/web.py
+++ b/omero/plugins/web.py
@@ -82,7 +82,7 @@ def wrapper(self, *args, **kwargs):
                 import django  # NOQA
             except Exception:
                 self.ctx.die(681, "ERROR: Django not installed!")
-            if django.VERSION < (1, 11) or django.VERSION >= (2, 0):
+            if django.VERSION < (3, 2) or django.VERSION >= (3, 3):
                 self.ctx.err(
                     "ERROR: Django version %s is not "
                     "supported!" % django.get_version()
diff --git a/omeroweb/api/views.py b/omeroweb/api/views.py
index 0451e9cd64..0649c5739c 100644
--- a/omeroweb/api/views.py
+++ b/omeroweb/api/views.py
@@ -22,7 +22,7 @@
 from django.views.generic import View
 from django.middleware import csrf
 from django.utils.decorators import method_decorator
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from . import api_settings
 
 import traceback
diff --git a/omeroweb/decorators.py b/omeroweb/decorators.py
index 4f7e971a7d..ad5ea65faf 100644
--- a/omeroweb/decorators.py
+++ b/omeroweb/decorators.py
@@ -33,7 +33,7 @@
 from django.conf import settings
 from django.utils.http import urlencode
 from functools import update_wrapper
-from django.core.urlresolvers import reverse, resolve, NoReverseMatch
+from django.urls import reverse, resolve, NoReverseMatch
 from django.core.cache import cache
 
 from omeroweb.utils import reverse_with_params
diff --git a/omeroweb/feedback/views.py b/omeroweb/feedback/views.py
index 488f460268..dd4bb6a9df 100644
--- a/omeroweb/feedback/views.py
+++ b/omeroweb/feedback/views.py
@@ -37,7 +37,7 @@
 from django.http import HttpResponse, HttpResponseRedirect, JsonResponse
 from django.http import HttpResponseServerError, HttpResponseNotFound
 from django.views.defaults import page_not_found
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from django.shortcuts import render
 
 from django.views.debug import get_exception_reporter_filter
diff --git a/omeroweb/filesessionstore.py b/omeroweb/filesessionstore.py
index 49dd7b75de..771101a398 100644
--- a/omeroweb/filesessionstore.py
+++ b/omeroweb/filesessionstore.py
@@ -92,7 +92,7 @@ def _expiry_date(self, session_data):
     def load(self):
         session_data = {}
         try:
-            with open(self._key_to_file(), "rb") as session_file:
+            with open(self._key_to_file(), "r") as session_file:
                 file_data = session_file.read()
             # Don't fail if there is no data in the session file.
             # We may have opened the empty placeholder file.
diff --git a/omeroweb/settings.py b/omeroweb/settings.py
index ec0cffade2..56590d8621 100644
--- a/omeroweb/settings.py
+++ b/omeroweb/settings.py
@@ -1366,8 +1366,9 @@ def process_custom_settings(
 
 
 def report_settings(module):
-    from django.views.debug import cleanse_setting
+    from django.views.debug import SafeExceptionReporterFilter
 
+    setting_filter = SafeExceptionReporterFilter()
     custom_settings_mappings = getattr(module, "CUSTOM_SETTINGS_MAPPINGS", {})
     for key in sorted(custom_settings_mappings):
         values = custom_settings_mappings[key]
@@ -1378,7 +1379,7 @@ def report_settings(module):
             logger.debug(
                 "%s = %r (source:%s)",
                 global_name,
-                cleanse_setting(global_name, global_value),
+                setting_filter.cleanse_setting(global_name, global_value),
                 source,
             )
 
@@ -1391,7 +1392,7 @@ def report_settings(module):
             logger.debug(
                 "%s = %r (deprecated:%s, %s)",
                 global_name,
-                cleanse_setting(global_name, global_value),
+                setting_filter.cleanse_setting(global_name, global_value),
                 key,
                 description,
             )
diff --git a/omeroweb/urls.py b/omeroweb/urls.py
index bfd65d19c8..5a930f897e 100755
--- a/omeroweb/urls.py
+++ b/omeroweb/urls.py
@@ -31,7 +31,7 @@
 from django.contrib.staticfiles.urls import staticfiles_urlpatterns
 from django.shortcuts import redirect
 
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from django.utils.functional import lazy
 from django.views.generic import RedirectView
 from django.views.decorators.cache import never_cache
@@ -101,7 +101,7 @@ def redirect_urlpatterns():
             if label == settings.OMEROWEB_ROOT_APPLICATION:
                 regex = r"^"
             else:
-                regex = "^(?i)%s/" % label
+                regex = "^%s/" % label
             urlpatterns.append(url(regex, include(urlmodule)))
         except ImportError:
             print(
@@ -120,12 +120,12 @@ def redirect_urlpatterns():
         r"^favicon\.ico$",
         lambda request: redirect("%s%s" % (settings.STATIC_URL, settings.FAVICON_URL)),
     ),
-    url(r"^(?i)webgateway/", include("omeroweb.webgateway.urls")),
-    url(r"^(?i)webadmin/", include("omeroweb.webadmin.urls")),
-    url(r"^(?i)webclient/", include("omeroweb.webclient.urls")),
-    url(r"^(?i)url/", include("omeroweb.webredirect.urls")),
-    url(r"^(?i)feedback/", include("omeroweb.feedback.urls")),
-    url(r"^(?i)api/", include("omeroweb.api.urls")),
+    url(r"^webgateway/", include("omeroweb.webgateway.urls")),
+    url(r"^webadmin/", include("omeroweb.webadmin.urls")),
+    url(r"^webclient/", include("omeroweb.webclient.urls")),
+    url(r"^url/", include("omeroweb.webredirect.urls")),
+    url(r"^feedback/", include("omeroweb.feedback.urls")),
+    url(r"^api/", include("omeroweb.api.urls")),
     url(r"^index/$", webclient_views.custom_index, name="webindex_custom"),
 ]
 
diff --git a/omeroweb/utils.py b/omeroweb/utils.py
index d16632be07..2764221992 100644
--- a/omeroweb/utils.py
+++ b/omeroweb/utils.py
@@ -22,8 +22,8 @@
 import logging
 
 from django.utils.http import urlencode
-from django.core.urlresolvers import reverse
-from django.core.urlresolvers import NoReverseMatch
+from django.urls import reverse
+from django.urls import NoReverseMatch
 from past.builtins import basestring
 
 
@@ -32,7 +32,7 @@
 
 def reverse_with_params(*args, **kwargs):
     """
-    Adds query string to django.core.urlresolvers.reverse
+    Adds query string to django.urls.reverse
     """
 
     url = ""
diff --git a/omeroweb/webadmin/custom_forms.py b/omeroweb/webadmin/custom_forms.py
index e9d9f64e0d..e404194cf4 100644
--- a/omeroweb/webadmin/custom_forms.py
+++ b/omeroweb/webadmin/custom_forms.py
@@ -27,12 +27,12 @@
 import re
 
 from django import forms
-from django.forms.fields import ChoiceField, EMPTY_VALUES
+from django.forms.fields import ChoiceField
 from django.forms.widgets import SelectMultiple, MultipleHiddenInput
 from django.forms import ModelChoiceField, ValidationError
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import smart_text
-from django.core.validators import validate_email
+from django.core.validators import validate_email, EMPTY_VALUES
 
 from past.builtins import long
 
@@ -76,7 +76,8 @@ def __iter__(self):
 
 
 class ServerModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -89,13 +90,11 @@ def _get_choices(self):
         # been consumed.
         return ServerQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's necessary
-        # because property() doesn't allow a subclass to overwrite only
-        # _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def to_python(self, value):
         if value in EMPTY_VALUES:
@@ -135,7 +134,8 @@ def __iter__(self):
 
 
 class GroupModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -148,13 +148,11 @@ def _get_choices(self):
         # been consumed.
         return GroupQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's necessary
-        # because property() doesn't allow a subclass to overwrite only
-        # _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def to_python(self, value):
         if value in EMPTY_VALUES:
@@ -320,7 +318,8 @@ def render(self, obj):
 
 
 class ExperimenterModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -333,13 +332,11 @@ def _get_choices(self):
         # been consumed.
         return ExperimenterQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's necessary
-        # because property() doesn't allow a subclass to overwrite only
-        # _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def to_python(self, value):
         """
diff --git a/omeroweb/webadmin/forms.py b/omeroweb/webadmin/forms.py
index 2e1deb453c..c30db16922 100644
--- a/omeroweb/webadmin/forms.py
+++ b/omeroweb/webadmin/forms.py
@@ -50,7 +50,7 @@ def __init__(self, *args, **kwargs):
         super(LoginForm, self).__init__(*args, **kwargs)
         self.fields["server"] = ServerModelChoiceField(Server, empty_label=None)
 
-        self.fields.keyOrder = ["server", "username", "password"]
+        self.field_order = ["server", "username", "password"]
 
     username = forms.CharField(
         max_length=50,
@@ -389,7 +389,7 @@ def __init__(
                 "title"
             ] = "Changing of system group name would be un-doable"
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "name",
             "description",
             "owners",
@@ -439,7 +439,7 @@ def __init__(self, *args, **kwargs):
                 queryset=kwargs["initial"]["experimenters"], required=False
             )
 
-        self.fields.keyOrder = ["owners", "members", "permissions"]
+        self.field_order = ["owners", "members", "permissions"]
 
     permissions = forms.ChoiceField(
         choices=PERMISSION_CHOICES,
@@ -465,7 +465,7 @@ def __init__(self, email_check=False, *args, **kwargs):
             self.fields["default_group"] = GroupModelChoiceField(
                 queryset=kwargs["initial"]["groups"], empty_label=None
             )
-        self.fields.keyOrder = [
+        self.field_order = [
             "omename",
             "first_name",
             "middle_name",
@@ -528,7 +528,7 @@ def __init__(self, *args, **kwargs):
                 queryset=kwargs["initial"]["experimenters"], required=False
             )
 
-        self.fields.keyOrder = ["members"]
+        self.field_order = ["members"]
 
 
 class UploadPhotoForm(forms.Form):
@@ -616,7 +616,7 @@ def __init__(self, entries, *args, **kwargs):
                     label=i + 1,
                 )
 
-        self.fields.keyOrder = [str(k) for k in self.fields.keys()]
+        self.field_order = [str(k) for k in self.fields.keys()]
 
 
 class EmailForm(forms.Form):
diff --git a/omeroweb/webadmin/urls.py b/omeroweb/webadmin/urls.py
index 05cb950a58..a880871d5b 100644
--- a/omeroweb/webadmin/urls.py
+++ b/omeroweb/webadmin/urls.py
@@ -47,12 +47,12 @@
     ),
     url(r"^groups/$", views.groups, name="wagroups"),
     url(
-        r"^group/(?P<action>((?i)new|create|edit|save))/" "(?:(?P<gid>[0-9]+)/)?$",
+        r"^group/(?P<action>(new|create|edit|save))/" "(?:(?P<gid>[0-9]+)/)?$",
         views.manage_group,
         name="wamanagegroupid",
     ),
     url(
-        r"^group_owner/(?P<action>((?i)edit|save))/(?P<gid>[0-9]+)/$",
+        r"^group_owner/(?P<action>(edit|save))/(?P<gid>[0-9]+)/$",
         views.manage_group_owner,
         name="wamanagegroupownerid",
     ),
diff --git a/omeroweb/webadmin/views.py b/omeroweb/webadmin/views.py
index 004296504c..882bae748f 100644
--- a/omeroweb/webadmin/views.py
+++ b/omeroweb/webadmin/views.py
@@ -38,7 +38,7 @@
 from omeroweb.version import omeroweb_buildyear as build_year
 from omeroweb.version import omeroweb_version as omero_version
 
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from django.views.decorators.debug import sensitive_post_parameters
 from django.http import HttpResponseRedirect
 from django.utils.translation import ugettext as _
diff --git a/omeroweb/webclient/custom_forms.py b/omeroweb/webclient/custom_forms.py
index 206f1522e0..58fcbe5435 100644
--- a/omeroweb/webclient/custom_forms.py
+++ b/omeroweb/webclient/custom_forms.py
@@ -24,10 +24,11 @@
 from django import forms
 from django.forms.widgets import SelectMultiple, MultipleHiddenInput
 
-from django.forms.fields import Field, EMPTY_VALUES
+from django.forms.fields import Field
 from django.forms import ModelChoiceField, ValidationError
 from django.utils.translation import ugettext_lazy as _
 from django.utils.encoding import smart_text
+from django.core.validators import EMPTY_VALUES
 
 from omero_model_FileAnnotationI import FileAnnotationI
 from omero_model_TagAnnotationI import TagAnnotationI
@@ -92,7 +93,8 @@ def __iter__(self):
 
 
 class MetadataModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -105,13 +107,11 @@ def _get_choices(self):
         # been consumed.
         return MetadataQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's necessary
-        # because property() doesn't allow a subclass to overwrite only
-        # _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def clean(self, value):
         Field.clean(self, value)
@@ -172,7 +172,8 @@ def __iter__(self):
 
 
 class AnnotationModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -185,13 +186,11 @@ def _get_choices(self):
         # been consumed.
         return AnnotationQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's
-        # necessary because property() doesn't allow a subclass to overwrite
-        # only _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def clean(self, value):
         Field.clean(self, value)
@@ -282,7 +281,8 @@ def __iter__(self):
 
 
 class ObjectModelChoiceField(ModelChoiceField):
-    def _get_choices(self):
+    @property
+    def choices(self):
         # If self._choices is set, then somebody must have manually set
         # the property self.choices. In this case, just return self._choices.
         if hasattr(self, "_choices"):
@@ -295,13 +295,11 @@ def _get_choices(self):
         # been consumed.
         return ObjectQuerySetIterator(self.queryset, self.empty_label)
 
-    def _set_choices(self, value):
-        # This method is copied from ChoiceField._set_choices(). It's
-        # necessary because property() doesn't allow a subclass to overwrite
-        # only _get_choices without implementing _set_choices.
-        self._choices = self.widget.choices = list(value)
+    def _set_queryset(self, queryset):
+        self._queryset = queryset
+        self.widget.choices = self.choices
 
-    choices = property(_get_choices, _set_choices)
+    queryset = property(ModelChoiceField._get_queryset, _set_queryset)
 
     def clean(self, value):
         Field.clean(self, value)
diff --git a/omeroweb/webclient/decorators.py b/omeroweb/webclient/decorators.py
index 7e7b170790..6be28c3b0a 100644
--- a/omeroweb/webclient/decorators.py
+++ b/omeroweb/webclient/decorators.py
@@ -30,8 +30,8 @@
 
 from django.http import HttpResponse
 from django.conf import settings
-from django.core.urlresolvers import reverse
-from django.core.urlresolvers import NoReverseMatch
+from django.urls import reverse
+from django.urls import NoReverseMatch
 
 from omeroweb.webclient.forms import GlobalSearchForm
 from omeroweb.utils import reverse_with_params
diff --git a/omeroweb/webclient/forms.py b/omeroweb/webclient/forms.py
index a8bcbfeecc..51ee515ec5 100644
--- a/omeroweb/webclient/forms.py
+++ b/omeroweb/webclient/forms.py
@@ -30,7 +30,7 @@
 from django.conf import settings
 from django import forms
 from django.forms.formsets import formset_factory
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 
 from omeroweb.custom_forms import NonASCIIForm
 from .custom_forms import MetadataModelChoiceField
@@ -88,7 +88,7 @@ def __init__(self, *args, **kwargs):
                 queryset=kwargs["initial"]["experimenters"],
                 widget=forms.SelectMultiple(attrs={"size": 28}),
             )
-        self.fields.keyOrder = [
+        self.field_order = [
             "message",
             "expiration",
             "enable",
@@ -382,7 +382,7 @@ def __init__(self, *args, **kwargs):
                     }
                 ),
             )
-        self.fields.keyOrder = ["active_group"]
+        self.field_order = ["active_group"]
 
 
 class WellIndexForm(forms.Form):
@@ -398,7 +398,7 @@ def __init__(self, *args, **kwargs):
                 }
             ),
         )
-        self.fields.keyOrder = ["index"]
+        self.field_order = ["index"]
 
 
 ###############################
@@ -805,7 +805,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["pockelCellSetting"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "name",
             "excitationWave",
             "emissionWave",
@@ -987,7 +987,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["lotNumber"])
 
-        self.fields.keyOrder = ["model", "manufacturer", "serialNumber", "lotNumber"]
+        self.field_order = ["model", "manufacturer", "serialNumber", "lotNumber"]
 
 
 class MetadataMicroscopeForm(forms.Form):
@@ -1195,7 +1195,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["type"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -1671,7 +1671,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["iris"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -1830,7 +1830,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["refractiveIndex"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -2312,7 +2312,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["transmittance"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -2797,7 +2797,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["binning"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -3400,7 +3400,7 @@ def __init__(self, *args, **kwargs):
             )
         set_widget_attrs(self.fields["attenuation"])
 
-        self.fields.keyOrder = [
+        self.field_order = [
             "model",
             "manufacturer",
             "serialNumber",
@@ -3589,7 +3589,7 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["co2percent"])
 
-        self.fields.keyOrder = ["airPressure", "co2percent", "humidity", "temperature"]
+        self.field_order = ["airPressure", "co2percent", "humidity", "temperature"]
 
 
 class MetadataStageLabelForm(forms.Form):
@@ -3724,4 +3724,4 @@ def __init__(self, *args, **kwargs):
             )
             set_widget_attrs(self.fields["positionz"])
 
-        self.fields.keyOrder = ["positionx", "positiony", "positionz"]
+        self.field_order = ["positionx", "positiony", "positionz"]
diff --git a/omeroweb/webclient/show.py b/omeroweb/webclient/show.py
index 428549b4be..7118ab8670 100644
--- a/omeroweb/webclient/show.py
+++ b/omeroweb/webclient/show.py
@@ -28,7 +28,7 @@
 import re
 
 from omero.rtypes import rint, rlong
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from copy import deepcopy
 from django.conf import settings
 
diff --git a/omeroweb/webclient/urls.py b/omeroweb/webclient/urls.py
index 2dadc4d6e8..9cc193d5d5 100644
--- a/omeroweb/webclient/urls.py
+++ b/omeroweb/webclient/urls.py
@@ -29,7 +29,7 @@
 from omeroweb.webclient import views
 from omeroweb.webgateway import views as webgateway
 from omeroweb.webclient.webclient_gateway import defaultThumbnail
-from django.core.urlresolvers import get_callable
+from django.urls import get_callable
 
 viewer_view = get_callable(settings.VIEWER_VIEW)
 
@@ -38,7 +38,7 @@
     url(r"^$", views.load_template, {"menu": "userdata"}, name="webindex"),
     # render main template
     url(
-        r"^(?P<menu>((?i)userdata|public|history|search|help|usertags))/$",
+        r"^(?P<menu>(userdata|public|history|search|help|usertags))/$",
         views.load_template,
         name="load_template",
     ),
@@ -65,7 +65,7 @@
     # loading data
     url(
         r"^load_plate/(?:(?P<o1_type>"
-        r"((?i)plate|acquisition))/)"
+        r"(plate|acquisition))/)"
         r"?(?:(?P<o1_id>[0-9]+)/)?$",
         views.load_plate,
         name="load_plate",
@@ -77,7 +77,7 @@
     ),  # Query E.g. ?Image=1,2&Dataset=3
     url(
         r"^load_chgrp_target/(?P<group_id>[0-9]+)/"
-        r"(?P<target_type>((?i)project|dataset|screen))/$",
+        r"(?P<target_type>(project|dataset|screen))/$",
         views.load_chgrp_target,
         name="load_chgrp_target",
     ),
@@ -94,7 +94,7 @@
     ),
     # load search
     url(
-        r"^load_searching/(?:(?P<form>((?i)form))/)?$",
+        r"^load_searching/(?:(?P<form>(form))/)?$",
         views.load_searching,
         name="load_searching",
     ),
@@ -112,7 +112,7 @@
         name="load_metadata_acquisition",
     ),
     url(
-        r"^metadata_preview/(?P<c_type>((?i)image|well))/"
+        r"^metadata_preview/(?P<c_type>(image|well))/"
         r"(?P<c_id>[0-9]+)/(?:(?P<share_id>[0-9]+)/)?$",
         views.load_metadata_preview,
         name="load_metadata_preview",
@@ -229,7 +229,7 @@
     # Fileset query (for delete or chgrp dialogs) obj-types and ids in REQUEST
     # data
     url(
-        r"^fileset_check/(?P<action>((?i)delete|chgrp))/$",
+        r"^fileset_check/(?P<action>(delete|chgrp))/$",
         views.fileset_check,
         name="fileset_check",
     ),
@@ -294,7 +294,7 @@
         name="download_orig_metadata",
     ),
     url(
-        r"^omero_table/(?P<file_id>[0-9]+)/(?:(?P<mtype>((?i)json|csv))/)?$",
+        r"^omero_table/(?P<file_id>[0-9]+)/(?:(?P<mtype>(json|csv))/)?$",
         views.omero_table,
         name="omero_table",
     ),
@@ -322,7 +322,7 @@
         name="download_original_file",
     ),  # for stderr, stdout etc
     url(
-        r"^figure_script/(?P<scriptName>" r"((?i)SplitView|Thumbnail|MakeMovie))/$",
+        r"^figure_script/(?P<scriptName>" r"(SplitView|Thumbnail|MakeMovie))/$",
         views.figure_script,
         name="figure_script",
     ),  # shows a form for running a script
diff --git a/omeroweb/webclient/views.py b/omeroweb/webclient/views.py
index a7f9d9a8fc..a9eb4f6378 100755
--- a/omeroweb/webclient/views.py
+++ b/omeroweb/webclient/views.py
@@ -62,7 +62,7 @@
 )
 from django.http import HttpResponseServerError, HttpResponseBadRequest
 from django.utils.http import urlencode
-from django.core.urlresolvers import reverse, NoReverseMatch
+from django.urls import reverse, NoReverseMatch
 from django.utils.encoding import smart_str
 from django.views.decorators.cache import never_cache
 from django.views.decorators.http import require_POST
diff --git a/omeroweb/webgateway/views.py b/omeroweb/webgateway/views.py
index 7280f68130..1835a21a14 100644
--- a/omeroweb/webgateway/views.py
+++ b/omeroweb/webgateway/views.py
@@ -40,7 +40,7 @@
 from django.views.decorators.http import require_POST
 from django.views.decorators.debug import sensitive_post_parameters
 from django.utils.decorators import method_decorator
-from django.core.urlresolvers import reverse, NoReverseMatch
+from django.urls import reverse, NoReverseMatch
 from django.conf import settings
 from wsgiref.util import FileWrapper
 from omero.rtypes import rlong, unwrap
@@ -49,7 +49,7 @@
 from .plategrid import PlateGrid
 from omeroweb.version import omeroweb_buildyear as build_year
 from .marshal import imageMarshal, shapeMarshal, rgb_int2rgba
-from django.contrib.staticfiles.templatetags.staticfiles import static
+from django.templatetags.static import static
 from django.views.generic import View
 from django.shortcuts import render
 from omeroweb.webadmin.forms import LoginForm
diff --git a/omeroweb/webredirect/views.py b/omeroweb/webredirect/views.py
index 2b11d31cb1..1319f787d2 100644
--- a/omeroweb/webredirect/views.py
+++ b/omeroweb/webredirect/views.py
@@ -29,7 +29,7 @@
 or anything."""
 
 from django.http import HttpResponseRedirect
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 from omeroweb.feedback.views import handlerInternalError
 import logging
 
diff --git a/setup.py b/setup.py
index 2c5329b409..d8d0f102ff 100755
--- a/setup.py
+++ b/setup.py
@@ -53,8 +53,8 @@ def read(fname):
         # requires Ice (use wheel for faster installs)
         "omero-py>=5.7.0",
         # minimum requirements for `omero web start`
-        "Django>=1.11,<2.0",
-        "django-pipeline==1.6.14",
+        "Django>=3.2,<4.0",
+        "django-pipeline==2.0.7",
         "gunicorn>=19.3",
         "omero-marshal>=0.7.0",
         "Pillow",


Test patch:
diff --git a/omeroweb/testlib/__init__.py b/omeroweb/testlib/__init__.py
index c9dced1fc0..80ac4ed313 100644
--- a/omeroweb/testlib/__init__.py
+++ b/omeroweb/testlib/__init__.py
@@ -28,7 +28,7 @@
 
 from django.test import Client
 from django.test.client import MULTIPART_CONTENT
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 
 from omero.testlib import ITest
 
diff --git a/test/unit/test_util.py b/test/unit/test_util.py
index e73f77874f..e9ee7ae574 100644
--- a/test/unit/test_util.py
+++ b/test/unit/test_util.py
@@ -25,7 +25,7 @@
 import pytest
 import json
 
-from django.core.urlresolvers import reverse
+from django.urls import reverse
 
 from omeroweb.utils import reverse_with_params, sort_properties_to_tuple
 from omeroweb.webclient.webclient_utils import formatPercentFraction