Repository: wagtail/wagtail

Issue:
Python 3.9 compatibility (resolve DeprecationWarning from collections)
### Issue Summary

Wagtail hits the warning `DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working` in various places.

- [x] Jinja2>2.10.1 
- [x] html5lib>1.0.1 (see https://github.com/html5lib/html5lib-python/issues/419)
- [x] beautifulsoup4>=4.8.0 (or maybe earlier)
- [x] fix wagtail/utils/l18n/translation.py:5 (#5485)

### Steps to Reproduce

1. run `tox -e py37-dj22-sqlite-noelasticsearch -- --deprecation all`
2. note deprecation warnings:


```
site-packages/jinja2/utils.py:485: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
site-packages/jinja2/runtime.py:318: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
```
resolved by https://github.com/pallets/jinja/pull/867 - the fix is on master not released yet (as of Jinja2==2.10.1)

```
site-packages/html5lib/_trie/_base.py:3: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  from collections import Mapping
```

Fixed in https://github.com/html5lib/html5lib-python/issues/402 , but not released yet as of 1.0.1 (see https://github.com/html5lib/html5lib-python/issues/419 )

```
site-packages/bs4/element.py:1134: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
  if not isinstance(formatter, collections.Callable):
```
https://bugs.launchpad.net/beautifulsoup/+bug/1778909 - resolved by beautifulsoup4>=4.8.0 (and earlier I think)

I'm also seeing one in Wagtail from my own project tests here, the above tox run didn't seem to hit it:

```
wagtail/utils/l18n/translation.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
```

See #5485 

* I have confirmed that this issue can be reproduced as described on a fresh Wagtail project: 

yes (on master at bb4e2fe2dfe69fc3143f38c6e34dbe6f2f2f01e0 )

### Technical details

* Python version: Run `python --version`.

Python 3.7.3

* Django version: Look in your requirements.txt, or run `pip show django | grep Version`.

Django 2.2

* Wagtail version: Look at the bottom of the Settings menu in the Wagtail admin, or run `pip show wagtail | grep Version:`.

2.7.0.alpha.0



Discussion:
Would it be a good idea to add python3.8 beta to the travis allowed failures?
@therefromhere Sounds good - have opened #5487 to add this.
@gasman Given that 2.7 is supposed to support Python 3.8, shouldn't we bump the lower bound of the libraries listed below?
By the way, it looks like deprecation in question was bumped from py3.8 to py3.9.

Using python:3.8 from dockerhub:

```
Python 3.8.0 (default, Oct 17 2019, 05:36:36) 
[GCC 8.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from collections import MutableMapping
<stdin>:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.9 it will stop working
```

So this ticket can probably either be closed or renamed to "Python 3.9..."

For the version numbers, I'm not sure the lower bounds need to change, since it doesn't matter unless you're running py3.9, but the upper bounds for some might need to.

Currently setup.py has beautifulsoup4>=4.5.1,<4.6.1 - I think the upper bound will need to increase?

Also html5lib, but there's not a fixed version released yet.

@loicteixeira I think it's a good thing to keep our dependency versions as wide as possible - the fact that we offer Python 3.8 support doesn't mean that we should lock down our dependencies to only the ones that are 3.8-compatible. At the extreme end, that means that we're restricting people to a version of Pillow that's only a few days old.

On one hand, it does mean that early adopters of Python 3.8 may have to figure out the required upgrades for themselves (hopefully nothing that a `pip install -U` can't solve); on the other hand, it means that people who are stuck on older versions of libraries for whatever reason (for example: they're running on a platform like Google AppEngine with limited ability to install server-level packages) aren't being left stranded.
I was wondering if we could set different install requirements depending on Python's version and it seems it should be possible with [environment markers](https://www.python.org/dev/peps/pep-0508/#environment-markers).

However it probably isn't worth as it will complexify the release process and might lead to other bugs (if done incorrectly or installed with incompatible versions of setuptools). And as you said, someone upgrading could easilly upgrade those dependencies.
Apparently the collections.abc change was deferred again to Python 3.10, but in any case, the final remaining deprecation warning was in botocore (a testing-only dependency), and version-bumping that has resolved it. Will open a PR shortly to make Python 3.9 support official (and current indications are that we'll be able to support 3.10 with no further changes too).

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -45,7 +45,7 @@
     'pytz>=2014.7',
     'elasticsearch>=1.0.0,<3.0',
     'Jinja2>=2.8,<3.0',
-    'boto3>=1.4,<1.5',
+    'boto3>=1.16,<1.17',
     'freezegun>=0.3.8',
     'openpyxl>=2.6.4',
     'Unidecode>=0.04.14,<2.0',
@@ -105,6 +105,7 @@
         'Programming Language :: Python :: 3.6',
         'Programming Language :: Python :: 3.7',
         'Programming Language :: Python :: 3.8',
+        'Programming Language :: Python :: 3.9',
         'Framework :: Django',
         'Framework :: Django :: 2.2',
         'Framework :: Django :: 3.0',


Test patch: