Repository: NatLibFi/Annif (language: python)

Issue:
Support Python 3.10
Ubuntu 22.04, which has been just released, comes with Python 3.10 by default. Annif should fully support Python 3.10.

[As reported in annif-users](https://groups.google.com/g/annif-users/c/suuql5d1ufs/m/-MmETFfrAAAJ), installing Omikuji and NN backends fail, and also installing Gensim and pycld3 dependencies have some issues:

## Omikuji
Current Omikuji does not have a wheel for Python 3.10, so installing Omikuji for Python 3.10 requires installing first cargo, but even after that, there is an error (at least on Docker image python:3.10-slim):
```
error: failed to parse manifest at `/tmp/pip-install-5yzjqez2/omikuji_cbdca1e9c9fb435e9d32fc4d65a5d33f/c-api/Cargo.toml`
Caused by:
     failed to parse the `edition` key
Caused by:
     this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.
 ```
I'm not sure if this can be handled some way or should it be fixed in Omikuji.

## NN ensemble
[Python 3.10 support requires TensorFlow 2.8 or later.](https://www.tensorflow.org/install/pip) Apparently due to NumPy version requirements.

## Gensim
Current Gensim releases have [wheels only for Python 3.6-3.9](https://pypi.org/project/gensim/#files), but the [next release will have wheel for Python 3.10 too](https://github.com/RaRe-Technologies/gensim/pull/3298). Before next release, for installing Annif on Python 3.10 it is required to have gcc etc. from build-essential package (which is the case for fastText anyway).

## pycld3
Also pycld3 has wheels only for Python 3.6-3.9, and installing pycld3 fails with
```
RuntimeError: The Protobuf compiler, `protoc`, which is required for building this package, could not be found.
```
Next release of pycld3 [should include wheel for Python 3.10](https://github.com/bsolomon1124/pycld3/issues/18).

## Other dependencies
Dropping Python 3.7 support  [would allow upgrading some pinned dependencies](https://github.com/NatLibFi/Annif/pull/549#issuecomment-1008662572).
Pin flake8 to version 4.x to avoid pytest-flake8 breakage
This PR fixes a recently surfaced [incompatibility](https://github.com/tholo/pytest-flake8/issues/87) that causes CI tests to fail. pytest-flake8 (version 1.1.1) is not compatible with flake8 version 5, released a few days ago. The PR pins the version of flake8 to 4.x, which fixes the issue for now but is probably not a good long term solution.

I think we need to reconsider the use of pytest-flake8. Many folks [seem](https://github.com/pypa/pep517/pull/147) [to](https://github.com/pytest-dev/pytest/issues/9217#issuecomment-965144375) [be](https://github.com/miketheman/pytest-socket/issues/82) abandoning this flaky (heh) integration and instead switching to a setup where flake8 is run outside pytest. I will open a separate issue about this.


Discussion / hints:
> ## Omikuji
> 
> Current Omikuji does not have a wheel for Python 3.10, so installing Omikuji for Python 3.10 requires installing first cargo, but even after that, there is an error (at least on Docker image python:3.10-slim):
> 
> ```
> error: failed to parse manifest at `/tmp/pip-install-5yzjqez2/omikuji_cbdca1e9c9fb435e9d32fc4d65a5d33f/c-api/Cargo.toml`
> Caused by:
>      failed to parse the `edition` key
> Caused by:
>      this version of Cargo is older than the `2021` edition, and only supports `2015` and `2018` editions.
> ```
> 
> I'm not sure if this can be handled some way or should it be fixed in Omikuji.

Now I managed in installing a newer version of cargo (1.60.0) which works with current Omikuji (the cargo version in apt repository was way too old). Downloading the installation script seems to be the [official way](https://www.rust-lang.org/tools/install), for example in Dockerfile:
```dockerfile
FROM python:3.10-slim-bullseye
RUN pip install --upgrade pip setuptools wheel
RUN apt update && apt install curl build-essential -y
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y 
ENV PATH="/root/.cargo/bin:${PATH}"
RUN pip install omikuji


Great that you found a solution, but I wonder if it would be better to ask upstream Omikuji to provide wheels for Python 3.10?
> Great that you found a solution, but I wonder if it would be better to ask upstream Omikuji to provide wheels for Python 3.10?

Yes that would be better. I'll make an issue.
Omikuji upstream now provides wheels for Python 3.10 as well, according to [this comment](https://github.com/tomtung/omikuji/issues/43#issuecomment-1129198884)


Fix patch:
diff --git a/.github/workflows/cicd.yml b/.github/workflows/cicd.yml
index 3dd468958..1d5aceefb 100644
--- a/.github/workflows/cicd.yml
+++ b/.github/workflows/cicd.yml
@@ -16,7 +16,7 @@ jobs:
     timeout-minutes: 10
     strategy:
       matrix:
-        python-version: [3.7, 3.8, 3.9]
+        python-version: ["3.8", "3.9", "3.10"]
     name: test on Python ${{ matrix.python-version }}
 
     steps:
@@ -39,22 +39,24 @@ jobs:
         python -m nltk.downloader punkt
         # Selectively install the optional dependencies for some Python versions
         # Install the optional neural network dependencies (TensorFlow and LMDB)
-        # - except for one Python version (3.8) so that we can test also without them
-        if [[ ${{ matrix.python-version }} != '3.8' ]]; then pip install .[nn]; fi
+        # - except for one Python version (3.9) so that we can test also without them
+        if [[ ${{ matrix.python-version }} != '3.9' ]]; then pip install .[nn]; fi
         # Install the optional Omikuji and YAKE dependencies
-        # - except for one Python version (3.8) so that we can test also without them
-        if [[ ${{ matrix.python-version }} != '3.8' ]]; then pip install .[omikuji,yake]; fi
-        # Install the optional fastText dependencies for Python 3.8 only
-        if [[ ${{ matrix.python-version }} == '3.8' ]]; then pip install .[fasttext]; fi
-        # Install the optional spaCy dependencies for Python 3.8 only
-        if [[ ${{ matrix.python-version }} == '3.8' ]]; then
+        # - except for one Python version (3.9) so that we can test also without them
+        if [[ ${{ matrix.python-version }} != '3.9' ]]; then pip install .[omikuji,yake]; fi
+        # Install the optional fastText dependencies for Python 3.9 only
+        if [[ ${{ matrix.python-version }} == '3.9' ]]; then pip install .[fasttext]; fi
+        # Install the optional spaCy dependencies for Python 3.9 only
+        if [[ ${{ matrix.python-version }} == '3.9' ]]; then
           pip install .[spacy]
           # download the small English pretrained spaCy model needed by spacy analyzer
           python -m spacy download en_core_web_sm --upgrade-strategy only-if-needed
         fi
-        # For Python 3.7
+        # For Python 3.8
         # - voikko and pycld3 dependencies
-        if [[ ${{ matrix.python-version }} == '3.7' ]]; then python -m pip install .[voikko,pycld3]; fi
+        if [[ ${{ matrix.python-version }} == '3.8' ]]; then python -m pip install .[voikko,pycld3]; fi
+        # Verify installed packages have compatible dependencies:
+        python -m pip check
 
     - name: Lint with flake8
       run: |
@@ -101,10 +103,10 @@ jobs:
     if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
     steps:
     - uses: actions/checkout@v3
-    - name: Set up Python 3.8
+    - name: Set up Python 3.9
       uses: actions/setup-python@v3
       with:
-        python-version: '3.8'
+        python-version: '3.9'
         cache: pip
         cache-dependency-path: setup.py
     - name: Build distribution
diff --git a/README.md b/README.md
index 871858e22..ab2e226f4 100644
--- a/README.md
+++ b/README.md
@@ -22,7 +22,7 @@ already functional for many common tasks.
 
 # Basic install
 
-You will need Python 3.7+ to install Annif.
+You will need Python 3.8+ to install Annif.
 
 The recommended way is to install Annif from
 [PyPI](https://pypi.org/project/annif/) into a virtual environment.
diff --git a/setup.py b/setup.py
index 6992c2eec..b80225591 100644
--- a/setup.py
+++ b/setup.py
@@ -22,9 +22,9 @@ def read(fname):
     packages=find_packages(),
     include_package_data=True,
     zip_safe=False,
-    python_requires='>=3.7',
+    python_requires='>=3.8',
     install_requires=[
-        'connexion[swagger-ui]==2.12.*',
+        'connexion[swagger-ui]==2.14.*',
         'swagger_ui_bundle',
         'flask>=1.0.4,<3',
         'flask-cors',
@@ -32,12 +32,12 @@ def read(fname):
         'click-log',
         'joblib==1.1.0',
         'nltk',
-        'gensim==4.1.*',
-        'scikit-learn==1.0.2',
-        'scipy==1.7.*',
+        'gensim==4.2.*',
+        'scikit-learn==1.1.1',
+        'scipy==1.8.*',
         'rdflib>=4.2,<7.0',
         'gunicorn',
-        'numpy==1.21.*',
+        'numpy==1.23.*',
         'optuna==2.10.*',
         'stwfsapy==0.3.*',
         'python-dateutil',
@@ -48,11 +48,11 @@ def read(fname):
     extras_require={
         'fasttext': ['fasttext==0.9.2'],
         'voikko': ['voikko'],
-        'nn': ['tensorflow-cpu==2.7.1', 'lmdb==1.3.0'],
+        'nn': ['tensorflow-cpu==2.9.1', 'lmdb==1.3.0'],
         'omikuji': ['omikuji==0.5.*'],
         'yake': ['yake==0.4.5'],
         'pycld3': ['pycld3'],
-        'spacy': ['spacy==3.2.*'],
+        'spacy': ['spacy==3.3.*'],
         'dev': [
             'codecov',
             'coverage<=6.2',
@@ -62,8 +62,7 @@ def read(fname):
             'pytest-flake8',
             'flake8<5',
             'bumpversion',
-            'autopep8',
-            'importlib_metadata'
+            'autopep8'
         ]
     },
     entry_points={


Test patch:
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 3d6ff8ca3..b6f7d439b 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -4,7 +4,7 @@
 import random
 import re
 import os.path
-import importlib_metadata
+import importlib
 import json
 from click.testing import CliRunner
 import annif.cli
@@ -806,5 +806,5 @@ def test_version_option():
         annif.cli.cli, ['--version'])
     assert not result.exception
     assert result.exit_code == 0
-    version = importlib_metadata.version('annif')
+    version = importlib.metadata.version('annif')
     assert result.output.strip() == version.strip()