Repository: NatLibFi/Annif (language: python)

Issue:
Use Black code style?
[Black](https://github.com/psf/black) seems to be in fashion. Adopting it would rid us of code style related worries.

If we decide to use it, we should consider using the following [integrations](https://black.readthedocs.io/en/stable/integrations/index.html):

* integrate Black with IDEs used by developers, when possible
* have a simple command to reformat all files with Black (or just check conformity)
* have a git pre-commit hook that performs Black reformatting (e.g. [this tutorial](https://towardsdatascience.com/keep-your-code-clean-using-black-pylint-git-hooks-pre-commit-baf6991f7376) which shows how to do it without using pre-commit.com, which would be an option too)
* have Black check the code under GitHub Actions CI

Black would replace autopep8 that we're currently using. There's still a need for flake8 checks, because flake8 checks other things than just pep8 code style.


Discussion / hints:

Fix patch:
diff --git a/.codeclimate.yml b/.codeclimate.yml
index 7de20d026..ae732bf8a 100644
--- a/.codeclimate.yml
+++ b/.codeclimate.yml
@@ -1,6 +1,11 @@
 engines:
   pep8:
     enabled: true
+    checks:
+      E203: # Check whitespace before ':'
+        enabled: false
+      E501: # Line length checks
+        enabled: false
   radon:
     enabled: true
   duplication:
diff --git a/.git-blame-ignore-revs b/.git-blame-ignore-revs
new file mode 100644
index 000000000..91c2ffe9c
--- /dev/null
+++ b/.git-blame-ignore-revs
@@ -0,0 +1,2 @@
+# Migrate code style to Black
+3bc18907354a40f1d89dca1833a2719ba7fb0933
diff --git a/.github/workflows/cicd.yml b/.github/workflows/cicd.yml
index afd52933c..dd67d6c8b 100644
--- a/.github/workflows/cicd.yml
+++ b/.github/workflows/cicd.yml
@@ -12,6 +12,14 @@ env:
   PIPX_BIN_DIR: "/home/runner/.local/bin"
   POETRY_VERSION: "1.2.0"
 jobs:
+
+  lint:
+    runs-on: ubuntu-latest
+    name: lint with Black
+    steps:
+      - uses: actions/checkout@v3
+      - uses: psf/black@6b42c2b8c9f9bd666120a2c19b8da509fe477f27
+
   test:
     runs-on: ubuntu-latest
     timeout-minutes: 15
@@ -23,6 +31,7 @@ jobs:
     steps:
     - uses: actions/checkout@v3
     - name: Install system packages
+
       run: |
         sudo apt-get install \
           libvoikko1 \
@@ -63,12 +72,6 @@ jobs:
         fi
         poetry run python -m nltk.downloader punkt
 
-    - name: Lint with flake8
-      run: |
-        # stop the build if there are Python syntax errors or undefined names
-        poetry run flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
-        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
-        poetry run flake8 . --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
     - name: Test with pytest
       run: |
         poetry run pytest --cov=./ --cov-report xml
@@ -77,7 +80,7 @@ jobs:
 
   publish-docker-latest:
     name: publish latest Docker image
-    needs: test
+    needs: [lint, test]
     runs-on: ubuntu-20.04
     timeout-minutes: 15
     if: github.event_name == 'push' && github.ref == 'refs/heads/master'
@@ -104,7 +107,7 @@ jobs:
 
   publish-release:
     name: publish release
-    needs: test
+    needs: [lint, test]
     runs-on: ubuntu-20.04
     if: github.event_name == 'push' && contains(github.ref, 'refs/tags/')
     steps:
diff --git a/README.md b/README.md
index 790283334..537aec15e 100644
--- a/README.md
+++ b/README.md
@@ -11,6 +11,7 @@
 [![LGTM: Python](https://img.shields.io/lgtm/grade/python/g/NatLibFi/Annif.svg?logo=lgtm&logoWidth=18)](https://lgtm.com/projects/g/NatLibFi/Annif/context:python)
 [![Quality Gate Status](https://sonarcloud.io/api/project_badges/measure?project=NatLibFi_Annif&metric=alert_status)](https://sonarcloud.io/dashboard?id=NatLibFi_Annif)
 [![docs](https://readthedocs.org/projects/annif/badge/?version=latest)](https://annif.readthedocs.io/en/latest/index.html)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
 Annif is an automated subject indexing toolkit. It was originally created as
 a statistical automated indexing tool that used metadata from the
@@ -99,6 +100,18 @@ Run `poetry shell` to enter the virtual environment and then run `pytest`.
 To have the test suite watch for changes in code and run automatically, use
 pytest-watch by running `ptw`.
 
+## Code style
+
+Annif code should follow the [Black style](https://black.readthedocs.io/en/stable/the_black_code_style/current_style.html).
+The Black tool is included as a developoment dependency; you can run `black .` in the project root to autoformat code. 
+You can set up a [pre-commit hook](https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks) to automate linting with Black and flake8 with every git commit by using the following in the file `.git/hooks/pre-commit`, which should have execute permission set:
+```bash
+#!/bin/sh
+
+black . --check --diff
+flake8
+```
+
 # Getting help
 
 Many resources are available:
diff --git a/annif/__init__.py b/annif/__init__.py
index d3ad2aa1d..8fe046b1a 100644
--- a/annif/__init__.py
+++ b/annif/__init__.py
@@ -6,7 +6,7 @@
 import connexion
 from flask_cors import CORS
 
-logger = logging.getLogger('annif')
+logger = logging.getLogger("annif")
 
 import annif.backend  # noqa
 
@@ -15,29 +15,30 @@ def create_app(config_name=None):
     # 'cxapp' here is the Connexion application that has a normal Flask app
     # as a property (cxapp.app)
 
-    specdir = os.path.join(os.path.dirname(__file__), 'swagger')
+    specdir = os.path.join(os.path.dirname(__file__), "swagger")
     cxapp = connexion.App(__name__, specification_dir=specdir)
     if config_name is None:
-        config_name = os.environ.get('ANNIF_CONFIG')
+        config_name = os.environ.get("ANNIF_CONFIG")
     if config_name is None:
-        if os.environ.get('FLASK_RUN_FROM_CLI') == 'true':
-            config_name = 'annif.default_config.Config'
+        if os.environ.get("FLASK_RUN_FROM_CLI") == "true":
+            config_name = "annif.default_config.Config"
         else:
-            config_name = 'annif.default_config.ProductionConfig'
-    logger.debug('creating app with configuration %s', config_name)
+            config_name = "annif.default_config.ProductionConfig"
+    logger.debug("creating app with configuration %s", config_name)
     cxapp.app.config.from_object(config_name)
-    cxapp.app.config.from_envvar('ANNIF_SETTINGS', silent=True)
+    cxapp.app.config.from_envvar("ANNIF_SETTINGS", silent=True)
 
-    cxapp.add_api('annif.yaml')
+    cxapp.add_api("annif.yaml")
 
     # add CORS support
     CORS(cxapp.app)
 
-    if cxapp.app.config['INITIALIZE_PROJECTS']:
+    if cxapp.app.config["INITIALIZE_PROJECTS"]:
         annif.registry.initialize_projects(cxapp.app)
 
     # register the views via blueprints
     from annif.views import bp
+
     cxapp.app.register_blueprint(bp)
 
     # return the Flask app
diff --git a/annif/analyzer/__init__.py b/annif/analyzer/__init__.py
index 954c315f6..fd4a3e908 100644
--- a/annif/analyzer/__init__.py
+++ b/annif/analyzer/__init__.py
@@ -15,10 +15,9 @@ def register_analyzer(analyzer):
 
 
 def get_analyzer(analyzerspec):
-    match = re.match(r'(\w+)(\((.*)\))?', analyzerspec)
+    match = re.match(r"(\w+)(\((.*)\))?", analyzerspec)
     if match is None:
-        raise ValueError(
-            "Invalid analyzer specification {}".format(analyzerspec))
+        raise ValueError("Invalid analyzer specification {}".format(analyzerspec))
 
     analyzer = match.group(1)
     posargs, kwargs = parse_args(match.group(3))
@@ -36,12 +35,14 @@ def get_analyzer(analyzerspec):
 # Optional analyzers
 try:
     from . import voikko
+
     register_analyzer(voikko.VoikkoAnalyzer)
 except ImportError:
     annif.logger.debug("voikko not available, not enabling voikko analyzer")
 
 try:
     from . import spacy
+
     register_analyzer(spacy.SpacyAnalyzer)
 except ImportError:
     annif.logger.debug("spaCy not available, not enabling spacy analyzer")
diff --git a/annif/analyzer/analyzer.py b/annif/analyzer/analyzer.py
index e83cf563a..37457069d 100644
--- a/annif/analyzer/analyzer.py
+++ b/annif/analyzer/analyzer.py
@@ -4,7 +4,7 @@
 import functools
 import unicodedata
 
-_KEY_TOKEN_MIN_LENGTH = 'token_min_length'
+_KEY_TOKEN_MIN_LENGTH = "token_min_length"
 
 
 class Analyzer(metaclass=abc.ABCMeta):
@@ -22,6 +22,7 @@ def __init__(self, **kwargs):
     def tokenize_sentences(self, text):
         """Tokenize a piece of text (e.g. a document) into sentences."""
         import nltk.tokenize
+
         return nltk.tokenize.sent_tokenize(text)
 
     @functools.lru_cache(maxsize=50000)
@@ -31,7 +32,7 @@ def is_valid_token(self, word):
             return False
         for char in word:
             category = unicodedata.category(char)
-            if category[0] == 'L':  # letter
+            if category[0] == "L":  # letter
                 return True
         return False
 
@@ -41,9 +42,12 @@ def tokenize_words(self, text, filter=True):
         punctuation, numbers or very short words)"""
 
         import nltk.tokenize
-        return [self._normalize_word(word)
-                for word in nltk.tokenize.word_tokenize(text)
-                if (not filter or self.is_valid_token(word))]
+
+        return [
+            self._normalize_word(word)
+            for word in nltk.tokenize.word_tokenize(text)
+            if (not filter or self.is_valid_token(word))
+        ]
 
     def _normalize_word(self, word):
         """Normalize (stem or lemmatize) a word form into a normal form."""
diff --git a/annif/analyzer/snowball.py b/annif/analyzer/snowball.py
index 88753297b..0e8e47150 100644
--- a/annif/analyzer/snowball.py
+++ b/annif/analyzer/snowball.py
@@ -10,6 +10,7 @@ class SnowballAnalyzer(analyzer.Analyzer):
     def __init__(self, param, **kwargs):
         self.param = param
         import nltk.stem.snowball
+
         self.stemmer = nltk.stem.snowball.SnowballStemmer(param)
         super().__init__(**kwargs)
 
diff --git a/annif/analyzer/spacy.py b/annif/analyzer/spacy.py
index 1e019a05f..014d41e52 100644
--- a/annif/analyzer/spacy.py
+++ b/annif/analyzer/spacy.py
@@ -4,7 +4,7 @@
 from annif.exception import OperationFailedException
 import annif.util
 
-_KEY_LOWERCASE = 'lowercase'
+_KEY_LOWERCASE = "lowercase"
 
 
 class SpacyAnalyzer(analyzer.Analyzer):
@@ -12,13 +12,15 @@ class SpacyAnalyzer(analyzer.Analyzer):
 
     def __init__(self, param, **kwargs):
         import spacy
+
         self.param = param
         try:
-            self.nlp = spacy.load(param, exclude=['ner', 'parser'])
+            self.nlp = spacy.load(param, exclude=["ner", "parser"])
         except IOError as err:
             raise OperationFailedException(
-                f"Loading spaCy model '{param}' failed - " +
-                f"please download the model.\n{err}")
+                f"Loading spaCy model '{param}' failed - "
+                + f"please download the model.\n{err}"
+            )
         if _KEY_LOWERCASE in kwargs:
             self.lowercase = annif.util.boolean(kwargs[_KEY_LOWERCASE])
         else:
@@ -26,10 +28,11 @@ def __init__(self, param, **kwargs):
         super().__init__(**kwargs)
 
     def tokenize_words(self, text, filter=True):
-        lemmas = [lemma
-                  for lemma in (token.lemma_
-                                for token in self.nlp(text.strip()))
-                  if (not filter or self.is_valid_token(lemma))]
+        lemmas = [
+            lemma
+            for lemma in (token.lemma_ for token in self.nlp(text.strip()))
+            if (not filter or self.is_valid_token(lemma))
+        ]
         if self.lowercase:
             return [lemma.lower() for lemma in lemmas]
         else:
diff --git a/annif/analyzer/voikko.py b/annif/analyzer/voikko.py
index c6173e248..4cd38b351 100644
--- a/annif/analyzer/voikko.py
+++ b/annif/analyzer/voikko.py
@@ -18,13 +18,13 @@ def __getstate__(self):
         instance is set to None because as a ctypes object it cannot be
         pickled."""
 
-        return {'param': self.param, 'voikko': None}
+        return {"param": self.param, "voikko": None}
 
     @functools.lru_cache(maxsize=500000)
     def _normalize_word(self, word):
         if self.voikko is None:
             self.voikko = voikko.libvoikko.Voikko(self.param)
         result = self.voikko.analyze(word)
-        if len(result) > 0 and 'BASEFORM' in result[0]:
-            return result[0]['BASEFORM']
+        if len(result) > 0 and "BASEFORM" in result[0]:
+            return result[0]["BASEFORM"]
         return word
diff --git a/annif/backend/__init__.py b/annif/backend/__init__.py
index 5ef28c9ac..a53913ae0 100644
--- a/annif/backend/__init__.py
+++ b/annif/backend/__init__.py
@@ -4,17 +4,20 @@
 # define functions for lazily importing each backend (alphabetical order)
 def _dummy():
     from . import dummy
+
     return dummy.DummyBackend
 
 
 def _ensemble():
     from . import ensemble
+
     return ensemble.EnsembleBackend
 
 
 def _fasttext():
     try:
         from . import fasttext
+
         return fasttext.FastTextBackend
     except ImportError:
         raise ValueError("fastText not available, cannot use fasttext backend")
@@ -22,26 +25,31 @@ def _fasttext():
 
 def _http():
     from . import http
+
     return http.HTTPBackend
 
 
 def _mllm():
     from . import mllm
+
     return mllm.MLLMBackend
 
 
 def _nn_ensemble():
     try:
         from . import nn_ensemble
+
         return nn_ensemble.NNEnsembleBackend
     except ImportError:
-        raise ValueError("Keras and TensorFlow not available, cannot use " +
-                         "nn_ensemble backend")
+        raise ValueError(
+            "Keras and TensorFlow not available, cannot use " + "nn_ensemble backend"
+        )
 
 
 def _omikuji():
     try:
         from . import omikuji
+
         return omikuji.OmikujiBackend
     except ImportError:
         raise ValueError("Omikuji not available, cannot use omikuji backend")
@@ -49,27 +57,32 @@ def _omikuji():
 
 def _pav():
     from . import pav
+
     return pav.PAVBackend
 
 
 def _stwfsa():
     from . import stwfsa
+
     return stwfsa.StwfsaBackend
 
 
 def _svc():
     from . import svc
+
     return svc.SVCBackend
 
 
 def _tfidf():
     from . import tfidf
+
     return tfidf.TFIDFBackend
 
 
 def _yake():
     try:
         from . import yake
+
         return yake.YakeBackend
     except ImportError:
         raise ValueError("YAKE not available, cannot use yake backend")
@@ -77,18 +90,18 @@ def _yake():
 
 # registry of the above functions
 _backend_fns = {
-    'dummy': _dummy,
-    'ensemble': _ensemble,
-    'fasttext': _fasttext,
-    'http': _http,
-    'mllm': _mllm,
-    'nn_ensemble': _nn_ensemble,
-    'omikuji': _omikuji,
-    'pav': _pav,
-    'stwfsa': _stwfsa,
-    'svc': _svc,
-    'tfidf': _tfidf,
-    'yake': _yake
+    "dummy": _dummy,
+    "ensemble": _ensemble,
+    "fasttext": _fasttext,
+    "http": _http,
+    "mllm": _mllm,
+    "nn_ensemble": _nn_ensemble,
+    "omikuji": _omikuji,
+    "pav": _pav,
+    "stwfsa": _stwfsa,
+    "svc": _svc,
+    "tfidf": _tfidf,
+    "yake": _yake,
 }
 
 
diff --git a/annif/backend/backend.py b/annif/backend/backend.py
index 97e89a44a..12e1cd927 100644
--- a/annif/backend/backend.py
+++ b/annif/backend/backend.py
@@ -13,7 +13,7 @@ class AnnifBackend(metaclass=abc.ABCMeta):
 
     name = None
 
-    DEFAULT_PARAMETERS = {'limit': 100}
+    DEFAULT_PARAMETERS = {"limit": 100}
 
     def __init__(self, backend_id, config_params, project):
         """Initialize backend with specific parameters. The
@@ -36,12 +36,14 @@ def params(self):
 
     @property
     def is_trained(self):
-        return bool(glob(os.path.join(self.datadir, '*')))
+        return bool(glob(os.path.join(self.datadir, "*")))
 
     @property
     def modification_time(self):
-        mtimes = [datetime.utcfromtimestamp(os.path.getmtime(p))
-                  for p in glob(os.path.join(self.datadir, '*'))]
+        mtimes = [
+            datetime.utcfromtimestamp(os.path.getmtime(p))
+            for p in glob(os.path.join(self.datadir, "*"))
+        ]
         most_recent = max(mtimes, default=None)
         if most_recent is None:
             return None
diff --git a/annif/backend/dummy.py b/annif/backend/dummy.py
index 04fc27497..83b9b6b53 100644
--- a/annif/backend/dummy.py
+++ b/annif/backend/dummy.py
@@ -19,17 +19,17 @@ def initialize(self, parallel=False):
         self.initialized = True
 
     def _suggest(self, text, params):
-        score = float(params.get('score', 1.0))
+        score = float(params.get("score", 1.0))
 
         # allow overriding returned subject via uri parameter
-        if 'uri' in params:
-            subject_id = self.project.subjects.by_uri(params['uri'])
+        if "uri" in params:
+            subject_id = self.project.subjects.by_uri(params["uri"])
         else:
             subject_id = self.subject_id
 
         return ListSuggestionResult(
-            [SubjectSuggestion(subject_id=subject_id,
-                               score=score)])
+            [SubjectSuggestion(subject_id=subject_id, score=score)]
+        )
 
     def _learn(self, corpus, params):
         # in this dummy backend we "learn" by picking up the subject ID
diff --git a/annif/backend/ensemble.py b/annif/backend/ensemble.py
index ea390ab8c..5c6590e46 100644
--- a/annif/backend/ensemble.py
+++ b/annif/backend/ensemble.py
@@ -15,14 +15,16 @@ class BaseEnsembleBackend(backend.AnnifBackend):
 
     def _get_sources_attribute(self, attr):
         params = self._get_backend_params(None)
-        sources = annif.util.parse_sources(params['sources'])
-        return [getattr(self.project.registry.get_project(project_id), attr)
-                for project_id, _ in sources]
+        sources = annif.util.parse_sources(params["sources"])
+        return [
+            getattr(self.project.registry.get_project(project_id), attr)
+            for project_id, _ in sources
+        ]
 
     def initialize(self, parallel=False):
         # initialize all the source projects
         params = self._get_backend_params(None)
-        for project_id, _ in annif.util.parse_sources(params['sources']):
+        for project_id, _ in annif.util.parse_sources(params["sources"]):
             project = self.project.registry.get_project(project_id)
             project.initialize(parallel)
 
@@ -37,27 +39,28 @@ def _suggest_with_sources(self, text, sources):
             source_project = self.project.registry.get_project(project_id)
             hits = source_project.suggest(text)
             self.debug(
-                'Got {} hits from project {}, weight {}'.format(
-                    len(hits), source_project.project_id, weight))
+                "Got {} hits from project {}, weight {}".format(
+                    len(hits), source_project.project_id, weight
+                )
+            )
             norm_hits = self._normalize_hits(hits, source_project)
             hits_from_sources.append(
                 annif.suggestion.WeightedSuggestion(
-                    hits=norm_hits,
-                    weight=weight,
-                    subjects=source_project.subjects))
+                    hits=norm_hits, weight=weight, subjects=source_project.subjects
+                )
+            )
         return hits_from_sources
 
     def _merge_hits_from_sources(self, hits_from_sources, params):
         """Hook for merging hits from sources. Can be overridden by
         subclasses."""
-        return annif.util.merge_hits(hits_from_sources,
-                                     len(self.project.subjects))
+        return annif.util.merge_hits(hits_from_sources, len(self.project.subjects))
 
     def _suggest(self, text, params):
-        sources = annif.util.parse_sources(params['sources'])
+        sources = annif.util.parse_sources(params["sources"])
         hits_from_sources = self._suggest_with_sources(text, sources)
         merged_hits = self._merge_hits_from_sources(hits_from_sources, params)
-        self.debug('{} hits after merging'.format(len(merged_hits)))
+        self.debug("{} hits after merging".format(len(merged_hits)))
         return merged_hits
 
 
@@ -66,9 +69,12 @@ class EnsembleOptimizer(hyperopt.HyperparameterOptimizer):
 
     def __init__(self, backend, corpus, metric):
         super().__init__(backend, corpus, metric)
-        self._sources = [project_id for project_id, _
-                         in annif.util.parse_sources(
-                             backend.config_params['sources'])]
+        self._sources = [
+            project_id
+            for project_id, _ in annif.util.parse_sources(
+                backend.config_params["sources"]
+            )
+        ]
 
     def _prepare(self, n_jobs=1):
         self._gold_subjects = []
@@ -82,14 +88,16 @@ def _prepare(self, n_jobs=1):
             self._backend.project.registry,
             self._sources,
             backend_params=None,
-            limit=int(self._backend.params['limit']),
-            threshold=0.0)
+            limit=int(self._backend.params["limit"]),
+            threshold=0.0,
+        )
 
         jobs, pool_class = annif.parallel.get_pool(n_jobs)
 
         with pool_class(jobs) as pool:
             for hits, subject_set in pool.imap_unordered(
-                    psmap.suggest, self._corpus.documents):
+                psmap.suggest, self._corpus.documents
+            ):
                 self._gold_subjects.append(subject_set)
                 self._source_hits.append(hits)
 
@@ -98,25 +106,32 @@ def _normalize(self, hps):
         return {source: hps[source] / total for source in hps}
 
     def _format_cfg_line(self, hps):
-        return 'sources=' + ','.join([f"{src}:{weight:.4f}"
-                                      for src, weight in hps.items()])
+        return "sources=" + ",".join(
+            [f"{src}:{weight:.4f}" for src, weight in hps.items()]
+        )
 
     def _objective(self, trial):
         batch = annif.eval.EvaluationBatch(self._backend.project.subjects)
-        weights = {project_id: trial.suggest_uniform(project_id, 0.0, 1.0)
-                   for project_id in self._sources}
+        weights = {
+            project_id: trial.suggest_uniform(project_id, 0.0, 1.0)
+            for project_id in self._sources
+        }
         for goldsubj, srchits in zip(self._gold_subjects, self._source_hits):
             weighted_hits = []
             for project_id, hits in srchits.items():
-                weighted_hits.append(annif.suggestion.WeightedSuggestion(
-                    hits=hits,
-                    weight=weights[project_id],
-                    subjects=self._backend.project.subjects))
+                weighted_hits.append(
+                    annif.suggestion.WeightedSuggestion(
+                        hits=hits,
+                        weight=weights[project_id],
+                        subjects=self._backend.project.subjects,
+                    )
+                )
             batch.evaluate(
                 annif.util.merge_hits(
-                    weighted_hits,
-                    len(self._backend.project.subjects)),
-                goldsubj)
+                    weighted_hits, len(self._backend.project.subjects)
+                ),
+                goldsubj,
+            )
         results = batch.results(metrics=[self._metric])
         return results[self._metric]
 
@@ -127,21 +142,21 @@ def _postprocess(self, study):
 
 class EnsembleBackend(BaseEnsembleBackend, hyperopt.AnnifHyperoptBackend):
     """Ensemble backend that combines results from multiple projects"""
+
     name = "ensemble"
 
     @property
     def is_trained(self):
-        sources_trained = self._get_sources_attribute('is_trained')
+        sources_trained = self._get_sources_attribute("is_trained")
         return all(sources_trained)
 
     @property
     def modification_time(self):
-        mtimes = self._get_sources_attribute('modification_time')
+        mtimes = self._get_sources_attribute("modification_time")
         return max(filter(None, mtimes), default=None)
 
     def get_hp_optimizer(self, corpus, metric):
         return EnsembleOptimizer(self, corpus, metric)
 
     def _train(self, corpus, params, jobs=0):
-        raise NotSupportedException(
-            'Training ensemble backend is not possible.')
+        raise NotSupportedException("Training ensemble backend is not possible.")
diff --git a/annif/backend/fasttext.py b/annif/backend/fasttext.py
index df1205c6c..7d7574c47 100644
--- a/annif/backend/fasttext.py
+++ b/annif/backend/fasttext.py
@@ -16,32 +16,32 @@ class FastTextBackend(mixins.ChunkingBackend, backend.AnnifBackend):
     name = "fasttext"
 
     FASTTEXT_PARAMS = {
-        'lr': float,
-        'lrUpdateRate': int,
-        'dim': int,
-        'ws': int,
-        'epoch': int,
-        'minCount': int,
-        'neg': int,
-        'wordNgrams': int,
-        'loss': str,
-        'bucket': int,
-        'minn': int,
-        'maxn': int,
-        'thread': int,
-        't': float,
-        'pretrainedVectors': str
+        "lr": float,
+        "lrUpdateRate": int,
+        "dim": int,
+        "ws": int,
+        "epoch": int,
+        "minCount": int,
+        "neg": int,
+        "wordNgrams": int,
+        "loss": str,
+        "bucket": int,
+        "minn": int,
+        "maxn": int,
+        "thread": int,
+        "t": float,
+        "pretrainedVectors": str,
     }
 
     DEFAULT_PARAMETERS = {
-        'dim': 100,
-        'lr': 0.25,
-        'epoch': 5,
-        'loss': 'hs',
+        "dim": 100,
+        "lr": 0.25,
+        "epoch": 5,
+        "loss": "hs",
     }
 
-    MODEL_FILE = 'fasttext-model'
-    TRAIN_FILE = 'fasttext-train.txt'
+    MODEL_FILE = "fasttext-model"
+    TRAIN_FILE = "fasttext-train.txt"
 
     # defaults for uninitialized instances
     _model = None
@@ -66,91 +66,99 @@ def _load_model(path):
     def initialize(self, parallel=False):
         if self._model is None:
             path = os.path.join(self.datadir, self.MODEL_FILE)
-            self.debug('loading fastText model from {}'.format(path))
+            self.debug("loading fastText model from {}".format(path))
             if os.path.exists(path):
                 self._model = self._load_model(path)
-                self.debug('loaded model {}'.format(str(self._model)))
-                self.debug('dim: {}'.format(self._model.get_dimension()))
+                self.debug("loaded model {}".format(str(self._model)))
+                self.debug("dim: {}".format(self._model.get_dimension()))
             else:
                 raise NotInitializedException(
-                    'model {} not found'.format(path),
-                    backend_id=self.backend_id)
+                    "model {} not found".format(path), backend_id=self.backend_id
+                )
 
     @staticmethod
     def _id_to_label(subject_id):
         return "__label__{:d}".format(subject_id)
 
     def _label_to_subject_id(self, label):
-        labelnum = label.replace('__label__', '')
+        labelnum = label.replace("__label__", "")
         return int(labelnum)
 
     def _write_train_file(self, corpus, filename):
-        with open(filename, 'w', encoding='utf-8') as trainfile:
+        with open(filename, "w", encoding="utf-8") as trainfile:
             for doc in corpus.documents:
                 text = self._normalize_text(doc.text)
-                if text == '':
+                if text == "":
                     continue
                 labels = [self._id_to_label(sid) for sid in doc.subject_set]
                 if labels:
-                    print(' '.join(labels), text, file=trainfile)
+                    print(" ".join(labels), text, file=trainfile)
                 else:
                     self.warning(f'no labels for document "{doc.text}"')
 
     def _normalize_text(self, text):
-        return ' '.join(self.project.analyzer.tokenize_words(text))
+        return " ".join(self.project.analyzer.tokenize_words(text))
 
     def _create_train_file(self, corpus):
-        self.info('creating fastText training file')
+        self.info("creating fastText training file")
 
-        annif.util.atomic_save(corpus,
-                               self.datadir,
-                               self.TRAIN_FILE,
-                               method=self._write_train_file)
+        annif.util.atomic_save(
+            corpus, self.datadir, self.TRAIN_FILE, method=self._write_train_file
+        )
 
     def _create_model(self, params, jobs):
-        self.info('creating fastText model')
+        self.info("creating fastText model")
         trainpath = os.path.join(self.datadir, self.TRAIN_FILE)
         modelpath = os.path.join(self.datadir, self.MODEL_FILE)
-        params = {param: self.FASTTEXT_PARAMS[param](val)
-                  for param, val in params.items()
-                  if param in self.FASTTEXT_PARAMS}
+        params = {
+            param: self.FASTTEXT_PARAMS[param](val)
+            for param, val in params.items()
+            if param in self.FASTTEXT_PARAMS
+        }
         if jobs != 0:  # jobs set by user to non-default value
-            params['thread'] = jobs
-        self.debug('Model parameters: {}'.format(params))
+            params["thread"] = jobs
+        self.debug("Model parameters: {}".format(params))
         self._model = fasttext.train_supervised(trainpath, **params)
         self._model.save_model(modelpath)
 
     def _train(self, corpus, params, jobs=0):
-        if corpus != 'cached':
+        if corpus != "cached":
             if corpus.is_empty():
                 raise NotSupportedException(
-                    'training backend {} with no documents' .format(
-                        self.backend_id))
+                    "training backend {} with no documents".format(self.backend_id)
+                )
             self._create_train_file(corpus)
         else:
             self.info("Reusing cached training data from previous run.")
         self._create_model(params, jobs)
 
     def _predict_chunks(self, chunktexts, limit):
-        return self._model.predict(list(
-            filter(None, [self._normalize_text(chunktext)
-                          for chunktext in chunktexts])), limit)
+        return self._model.predict(
+            list(
+                filter(
+                    None, [self._normalize_text(chunktext) for chunktext in chunktexts]
+                )
+            ),
+            limit,
+        )
 
     def _suggest_chunks(self, chunktexts, params):
-        limit = int(params['limit'])
-        chunklabels, chunkscores = self._predict_chunks(
-            chunktexts, limit)
+        limit = int(params["limit"])
+        chunklabels, chunkscores = self._predict_chunks(chunktexts, limit)
         label_scores = collections.defaultdict(float)
         for labels, scores in zip(chunklabels, chunkscores):
             for label, score in zip(labels, scores):
                 label_scores[label] += score
-        best_labels = sorted([(score, label)
-                              for label, score in label_scores.items()],
-                             reverse=True)
+        best_labels = sorted(
+            [(score, label) for label, score in label_scores.items()], reverse=True
+        )
 
         results = []
         for score, label in best_labels[:limit]:
-            results.append(SubjectSuggestion(
-                subject_id=self._label_to_subject_id(label),
-                score=score / len(chunktexts)))
+            results.append(
+                SubjectSuggestion(
+                    subject_id=self._label_to_subject_id(label),
+                    score=score / len(chunktexts),
+                )
+            )
         return ListSuggestionResult(results)
diff --git a/annif/backend/http.py b/annif/backend/http.py
index 52ff9da81..71b1f6eb4 100644
--- a/annif/backend/http.py
+++ b/annif/backend/http.py
@@ -15,11 +15,11 @@ class HTTPBackend(backend.AnnifBackend):
 
     @property
     def is_trained(self):
-        return self._get_project_info('is_trained')
+        return self._get_project_info("is_trained")
 
     @property
     def modification_time(self):
-        mtime = self._get_project_info('modification_time')
+        mtime = self._get_project_info("modification_time")
         if mtime is None:
             return None
         return dateutil.parser.parse(mtime)
@@ -27,7 +27,7 @@ def modification_time(self):
     def _get_project_info(self, key):
         params = self._get_backend_params(None)
         try:
-            req = requests.get(params['endpoint'].replace('/suggest', ''))
+            req = requests.get(params["endpoint"].replace("/suggest", ""))
             req.raise_for_status()
         except requests.exceptions.RequestException as err:
             msg = f"HTTP request failed: {err}"
@@ -44,12 +44,12 @@ def _get_project_info(self, key):
             return None
 
     def _suggest(self, text, params):
-        data = {'text': text}
-        if 'project' in params:
-            data['project'] = params['project']
+        data = {"text": text}
+        if "project" in params:
+            data["project"] = params["project"]
 
         try:
-            req = requests.post(params['endpoint'], data=data)
+            req = requests.post(params["endpoint"], data=data)
             req.raise_for_status()
         except requests.exceptions.RequestException as err:
             self.warning("HTTP request failed: {}".format(err))
@@ -61,16 +61,20 @@ def _suggest(self, text, params):
             self.warning("JSON decode failed: {}".format(err))
             return ListSuggestionResult([])
 
-        if 'results' in response:
-            results = response['results']
+        if "results" in response:
+            results = response["results"]
         else:
             results = response
 
         try:
-            subject_suggestions = [SubjectSuggestion(
-                subject_id=self.project.subjects.by_uri(hit['uri']),
-                score=hit['score'])
-                for hit in results if hit['score'] > 0.0]
+            subject_suggestions = [
+                SubjectSuggestion(
+                    subject_id=self.project.subjects.by_uri(hit["uri"]),
+                    score=hit["score"],
+                )
+                for hit in results
+                if hit["score"] > 0.0
+            ]
         except (TypeError, ValueError) as err:
             self.warning("Problem interpreting JSON data: {}".format(err))
             return ListSuggestionResult([])
diff --git a/annif/backend/hyperopt.py b/annif/backend/hyperopt.py
index 8448186be..ec261f8d0 100644
--- a/annif/backend/hyperopt.py
+++ b/annif/backend/hyperopt.py
@@ -8,7 +8,7 @@
 from .backend import AnnifBackend
 
 
-HPRecommendation = collections.namedtuple('HPRecommendation', 'lines score')
+HPRecommendation = collections.namedtuple("HPRecommendation", "lines score")
 
 
 class TrialWriter:
@@ -26,12 +26,18 @@ def write(self, study, trial):
 
         if not self.header_written:
             param_names = list(trial.params.keys())
-            print('\t'.join(['trial', 'value'] + param_names),
-                  file=self.results_file)
+            print("\t".join(["trial", "value"] + param_names), file=self.results_file)
             self.header_written = True
-        print('\t'.join((str(e) for e in [trial.number, trial.value] +
-                         list(self.normalize_func(trial.params).values()))),
-              file=self.results_file)
+        print(
+            "\t".join(
+                (
+                    str(e)
+                    for e in [trial.number, trial.value]
+                    + list(self.normalize_func(trial.params).values())
+                )
+            ),
+            file=self.results_file,
+        )
 
 
 class HyperparameterOptimizer:
@@ -75,16 +81,19 @@ def optimize(self, n_trials, n_jobs, results_file):
         else:
             callbacks = []
 
-        study = optuna.create_study(direction='maximize')
+        study = optuna.create_study(direction="maximize")
         # silence the ExperimentalWarning when using the Optuna progress bar
-        warnings.filterwarnings("ignore",
-                                category=optuna.exceptions.ExperimentalWarning)
-        study.optimize(self._objective,
-                       n_trials=n_trials,
-                       n_jobs=n_jobs,
-                       callbacks=callbacks,
-                       gc_after_trial=False,
-                       show_progress_bar=(n_jobs == 1))
+        warnings.filterwarnings(
+            "ignore", category=optuna.exceptions.ExperimentalWarning
+        )
+        study.optimize(
+            self._objective,
+            n_trials=n_trials,
+            n_jobs=n_jobs,
+            callbacks=callbacks,
+            gc_after_trial=False,
+            show_progress_bar=(n_jobs == 1),
+        )
         return self._postprocess(study)
 
 
diff --git a/annif/backend/mixins.py b/annif/backend/mixins.py
index 98724b662..ccccb13fd 100644
--- a/annif/backend/mixins.py
+++ b/annif/backend/mixins.py
@@ -13,7 +13,7 @@
 class ChunkingBackend(metaclass=abc.ABCMeta):
     """Annif backend mixin that implements chunking of input"""
 
-    DEFAULT_PARAMETERS = {'chunksize': 1}
+    DEFAULT_PARAMETERS = {"chunksize": 1}
 
     def default_params(self):
         return self.DEFAULT_PARAMETERS
@@ -26,15 +26,16 @@ def _suggest_chunks(self, chunktexts, params):
         pass  # pragma: no cover
 
     def _suggest(self, text, params):
-        self.debug('Suggesting subjects for text "{}..." (len={})'.format(
-            text[:20], len(text)))
+        self.debug(
+            'Suggesting subjects for text "{}..." (len={})'.format(text[:20], len(text))
+        )
         sentences = self.project.analyzer.tokenize_sentences(text)
-        self.debug('Found {} sentences'.format(len(sentences)))
-        chunksize = int(params['chunksize'])
+        self.debug("Found {} sentences".format(len(sentences)))
+        chunksize = int(params["chunksize"])
         chunktexts = []
         for i in range(0, len(sentences), chunksize):
-            chunktexts.append(' '.join(sentences[i:i + chunksize]))
-        self.debug('Split sentences into {} chunks'.format(len(chunktexts)))
+            chunktexts.append(" ".join(sentences[i : i + chunksize]))
+        self.debug("Split sentences into {} chunks".format(len(chunktexts)))
         if len(chunktexts) == 0:  # no input, empty result
             return ListSuggestionResult([])
         return self._suggest_chunks(chunktexts, params)
@@ -43,7 +44,7 @@ def _suggest(self, text, params):
 class TfidfVectorizerMixin:
     """Annif backend mixin that implements TfidfVectorizer functionality"""
 
-    VECTORIZER_FILE = 'vectorizer'
+    VECTORIZER_FILE = "vectorizer"
 
     vectorizer = None
 
@@ -51,20 +52,19 @@ def initialize_vectorizer(self):
         if self.vectorizer is None:
             path = os.path.join(self.datadir, self.VECTORIZER_FILE)
             if os.path.exists(path):
-                self.debug('loading vectorizer from {}'.format(path))
+                self.debug("loading vectorizer from {}".format(path))
                 self.vectorizer = joblib.load(path)
             else:
                 raise NotInitializedException(
                     "vectorizer file '{}' not found".format(path),
-                    backend_id=self.backend_id)
+                    backend_id=self.backend_id,
+                )
 
     def create_vectorizer(self, input, params={}):
-        self.info('creating vectorizer')
+        self.info("creating vectorizer")
         self.vectorizer = TfidfVectorizer(**params)
         veccorpus = self.vectorizer.fit_transform(input)
         annif.util.atomic_save(
-            self.vectorizer,
-            self.datadir,
-            self.VECTORIZER_FILE,
-            method=joblib.dump)
+            self.vectorizer, self.datadir, self.VECTORIZER_FILE, method=joblib.dump
+        )
         return veccorpus
diff --git a/annif/backend/mllm.py b/annif/backend/mllm.py
index 216164d07..f55e5e8cf 100644
--- a/annif/backend/mllm.py
+++ b/annif/backend/mllm.py
@@ -30,10 +30,10 @@ def _prepare(self, n_jobs=1):
 
     def _objective(self, trial):
         params = {
-            'min_samples_leaf': trial.suggest_int('min_samples_leaf', 5, 30),
-            'max_leaf_nodes': trial.suggest_int('max_leaf_nodes', 100, 2000),
-            'max_samples': trial.suggest_float('max_samples', 0.5, 1.0),
-            'limit': 100
+            "min_samples_leaf": trial.suggest_int("min_samples_leaf", 5, 30),
+            "max_leaf_nodes": trial.suggest_int("max_leaf_nodes", 100, 2000),
+            "max_samples": trial.suggest_float("max_samples", 0.5, 1.0),
+            "limit": 100,
         }
         model = self._backend._model._create_classifier(params)
         model.fit(self._train_x, self._train_y)
@@ -41,11 +41,9 @@ def _objective(self, trial):
         batch = annif.eval.EvaluationBatch(self._backend.project.subjects)
         for goldsubj, candidates in zip(self._gold_subjects, self._candidates):
             if candidates:
-                features = \
-                    self._backend._model._candidates_to_features(candidates)
+                features = self._backend._model._candidates_to_features(candidates)
                 scores = model.predict_proba(features)
-                ranking = self._backend._model._prediction_to_list(
-                    scores, candidates)
+                ranking = self._backend._model._prediction_to_list(scores, candidates)
             else:
                 ranking = []
             results = self._backend._prediction_to_result(ranking, params)
@@ -58,26 +56,27 @@ def _postprocess(self, study):
         lines = [
             f"min_samples_leaf={bp['min_samples_leaf']}",
             f"max_leaf_nodes={bp['max_leaf_nodes']}",
-            f"max_samples={bp['max_samples']:.4f}"
+            f"max_samples={bp['max_samples']:.4f}",
         ]
         return hyperopt.HPRecommendation(lines=lines, score=study.best_value)
 
 
 class MLLMBackend(hyperopt.AnnifHyperoptBackend):
     """Maui-like Lexical Matching backend for Annif"""
+
     name = "mllm"
 
     # defaults for unitialized instances
     _model = None
 
-    MODEL_FILE = 'mllm-model.gz'
-    TRAIN_FILE = 'mllm-train.gz'
+    MODEL_FILE = "mllm-model.gz"
+    TRAIN_FILE = "mllm-train.gz"
 
     DEFAULT_PARAMETERS = {
-        'min_samples_leaf': 20,
-        'max_leaf_nodes': 1000,
-        'max_samples': 0.9,
-        'use_hidden_labels': False
+        "min_samples_leaf": 20,
+        "max_leaf_nodes": 1000,
+        "max_samples": 0.9,
+        "use_hidden_labels": False,
     }
 
     def get_hp_optimizer(self, corpus, metric):
@@ -90,13 +89,13 @@ def default_params(self):
 
     def _load_model(self):
         path = os.path.join(self.datadir, self.MODEL_FILE)
-        self.debug('loading model from {}'.format(path))
+        self.debug("loading model from {}".format(path))
         if os.path.exists(path):
             return MLLMModel.load(path)
         else:
             raise NotInitializedException(
-                'model {} not found'.format(path),
-                backend_id=self.backend_id)
+                "model {} not found".format(path), backend_id=self.backend_id
+            )
 
     def _load_train_data(self):
         path = os.path.join(self.datadir, self.TRAIN_FILE)
@@ -104,31 +103,28 @@ def _load_train_data(self):
             return joblib.load(path)
         else:
             raise NotInitializedException(
-                'train data file {} not found'.format(path),
-                backend_id=self.backend_id)
+                "train data file {} not found".format(path), backend_id=self.backend_id
+            )
 
     def initialize(self, parallel=False):
         if self._model is None:
             self._model = self._load_model()
 
     def _train(self, corpus, params, jobs=0):
-        self.info('starting train')
-        if corpus != 'cached':
+        self.info("starting train")
+        if corpus != "cached":
             if corpus.is_empty():
                 raise NotSupportedException(
-                    'training backend {} with no documents' .format(
-                        self.backend_id))
+                    "training backend {} with no documents".format(self.backend_id)
+                )
             self.info("preparing training data")
             self._model = MLLMModel()
-            train_data = self._model.prepare_train(corpus,
-                                                   self.project.vocab,
-                                                   self.project.analyzer,
-                                                   params,
-                                                   jobs)
-            annif.util.atomic_save(train_data,
-                                   self.datadir,
-                                   self.TRAIN_FILE,
-                                   method=joblib.dump)
+            train_data = self._model.prepare_train(
+                corpus, self.project.vocab, self.project.analyzer, params, jobs
+            )
+            annif.util.atomic_save(
+                train_data, self.datadir, self.TRAIN_FILE, method=joblib.dump
+            )
         else:
             self.info("reusing cached training data from previous run")
             self._model = self._load_model()
@@ -137,11 +133,8 @@ def _train(self, corpus, params, jobs=0):
         self.info("training model")
         self._model.train(train_data[0], train_data[1], params)
 
-        self.info('saving model')
-        annif.util.atomic_save(
-            self._model,
-            self.datadir,
-            self.MODEL_FILE)
+        self.info("saving model")
+        annif.util.atomic_save(self._model, self.datadir, self.MODEL_FILE)
 
     def _generate_candidates(self, text):
         return self._model.generate_candidates(text, self.project.analyzer)
@@ -151,8 +144,7 @@ def _prediction_to_result(self, prediction, params):
         for score, subject_id in prediction:
             vector[subject_id] = score
         result = VectorSuggestionResult(vector)
-        return result.filter(self.project.subjects,
-                             limit=int(params['limit']))
+        return result.filter(self.project.subjects, limit=int(params["limit"]))
 
     def _suggest(self, text, params):
         candidates = self._generate_candidates(text)
diff --git a/annif/backend/nn_ensemble.py b/annif/backend/nn_ensemble.py
index 1b195a989..219ddc5f1 100644
--- a/annif/backend/nn_ensemble.py
+++ b/annif/backend/nn_ensemble.py
@@ -24,7 +24,7 @@
 
 def idx_to_key(idx):
     """convert an integer index to a binary key for use in LMDB"""
-    return b'%08d' % idx
+    return b"%08d" % idx
 
 
 def key_to_idx(key):
@@ -78,28 +78,27 @@ def __len__(self):
 
 class MeanLayer(Layer):
     """Custom Keras layer that calculates mean values along the 2nd axis."""
+
     def call(self, inputs):
         return K.mean(inputs, axis=2)
 
 
-class NNEnsembleBackend(
-        backend.AnnifLearningBackend,
-        ensemble.BaseEnsembleBackend):
+class NNEnsembleBackend(backend.AnnifLearningBackend, ensemble.BaseEnsembleBackend):
     """Neural network ensemble backend that combines results from multiple
     projects"""
 
     name = "nn_ensemble"
 
     MODEL_FILE = "nn-model.h5"
-    LMDB_FILE = 'nn-train.mdb'
+    LMDB_FILE = "nn-train.mdb"
 
     DEFAULT_PARAMETERS = {
-        'nodes': 100,
-        'dropout_rate': 0.2,
-        'optimizer': 'adam',
-        'epochs': 10,
-        'learn-epochs': 1,
-        'lmdb_map_size': 1024 * 1024 * 1024
+        "nodes": 100,
+        "dropout_rate": 0.2,
+        "optimizer": "adam",
+        "epochs": 10,
+        "learn-epochs": 1,
+        "lmdb_map_size": 1024 * 1024 * 1024,
     }
 
     # defaults for uninitialized instances
@@ -121,20 +120,25 @@ def initialize(self, parallel=False):
         model_filename = os.path.join(self.datadir, self.MODEL_FILE)
         if not os.path.exists(model_filename):
             raise NotInitializedException(
-                'model file {} not found'.format(model_filename),
-                backend_id=self.backend_id)
-        self.debug('loading Keras model from {}'.format(model_filename))
-        self._model = load_model(model_filename,
-                                 custom_objects={'MeanLayer': MeanLayer})
+                "model file {} not found".format(model_filename),
+                backend_id=self.backend_id,
+            )
+        self.debug("loading Keras model from {}".format(model_filename))
+        self._model = load_model(
+            model_filename, custom_objects={"MeanLayer": MeanLayer}
+        )
 
     def _merge_hits_from_sources(self, hits_from_sources, params):
-        score_vector = np.array([np.sqrt(hits.as_vector(len(subjects)))
-                                 * weight * len(hits_from_sources)
-                                 for hits, weight, subjects
-                                 in hits_from_sources],
-                                dtype=np.float32)
+        score_vector = np.array(
+            [
+                np.sqrt(hits.as_vector(len(subjects))) * weight * len(hits_from_sources)
+                for hits, weight, subjects in hits_from_sources
+            ],
+            dtype=np.float32,
+        )
         results = self._model.predict(
-            np.expand_dims(score_vector.transpose(), 0), verbose=0)
+            np.expand_dims(score_vector.transpose(), 0), verbose=0
+        )
         return VectorSuggestionResult(results[0])
 
     def _create_model(self, sources):
@@ -143,49 +147,48 @@ def _create_model(self, sources):
         inputs = Input(shape=(len(self.project.subjects), len(sources)))
 
         flat_input = Flatten()(inputs)
-        drop_input = Dropout(
-            rate=float(
-                self.params['dropout_rate']))(flat_input)
-        hidden = Dense(int(self.params['nodes']),
-                       activation="relu")(drop_input)
-        drop_hidden = Dropout(rate=float(self.params['dropout_rate']))(hidden)
-        delta = Dense(len(self.project.subjects),
-                      kernel_initializer='zeros',
-                      bias_initializer='zeros')(drop_hidden)
+        drop_input = Dropout(rate=float(self.params["dropout_rate"]))(flat_input)
+        hidden = Dense(int(self.params["nodes"]), activation="relu")(drop_input)
+        drop_hidden = Dropout(rate=float(self.params["dropout_rate"]))(hidden)
+        delta = Dense(
+            len(self.project.subjects),
+            kernel_initializer="zeros",
+            bias_initializer="zeros",
+        )(drop_hidden)
 
         mean = MeanLayer()(inputs)
 
         predictions = Add()([mean, delta])
 
         self._model = Model(inputs=inputs, outputs=predictions)
-        self._model.compile(optimizer=self.params['optimizer'],
-                            loss='binary_crossentropy',
-                            metrics=['top_k_categorical_accuracy'])
-        if 'lr' in self.params:
-            self._model.optimizer.learning_rate.assign(
-                float(self.params['lr']))
+        self._model.compile(
+            optimizer=self.params["optimizer"],
+            loss="binary_crossentropy",
+            metrics=["top_k_categorical_accuracy"],
+        )
+        if "lr" in self.params:
+            self._model.optimizer.learning_rate.assign(float(self.params["lr"]))
 
         summary = []
         self._model.summary(print_fn=summary.append)
         self.debug("Created model: \n" + "\n".join(summary))
 
     def _train(self, corpus, params, jobs=0):
-        sources = annif.util.parse_sources(self.params['sources'])
+        sources = annif.util.parse_sources(self.params["sources"])
         self._create_model(sources)
         self._fit_model(
             corpus,
-            epochs=int(params['epochs']),
-            lmdb_map_size=int(params['lmdb_map_size']),
-            n_jobs=jobs)
+            epochs=int(params["epochs"]),
+            lmdb_map_size=int(params["lmdb_map_size"]),
+            n_jobs=jobs,
+        )
 
     def _corpus_to_vectors(self, corpus, seq, n_jobs):
         # pass corpus through all source projects
-        sources = dict(
-            annif.util.parse_sources(self.params['sources']))
+        sources = dict(annif.util.parse_sources(self.params["sources"]))
 
         # initialize the source projects before forking, to save memory
-        self.info(
-            f"Initializing source projects: {', '.join(sources.keys())}")
+        self.info(f"Initializing source projects: {', '.join(sources.keys())}")
         for project_id in sources.keys():
             project = self.project.registry.get_project(project_id)
             project.initialize(parallel=True)
@@ -195,24 +198,24 @@ def _corpus_to_vectors(self, corpus, seq, n_jobs):
             list(sources.keys()),
             backend_params=None,
             limit=None,
-            threshold=0.0)
+            threshold=0.0,
+        )
 
         jobs, pool_class = annif.parallel.get_pool(n_jobs)
 
         self.info("Processing training documents...")
         with pool_class(jobs) as pool:
             for hits, subject_set in pool.imap_unordered(
-                    psmap.suggest, corpus.documents):
+                psmap.suggest, corpus.documents
+            ):
                 doc_scores = []
                 for project_id, p_hits in hits.items():
                     vector = p_hits.as_vector(len(self.project.subjects))
-                    doc_scores.append(np.sqrt(vector)
-                                      * sources[project_id]
-                                      * len(sources))
-                score_vector = np.array(doc_scores,
-                                        dtype=np.float32).transpose()
-                true_vector = subject_set.as_vector(
-                    len(self.project.subjects))
+                    doc_scores.append(
+                        np.sqrt(vector) * sources[project_id] * len(sources)
+                    )
+                score_vector = np.array(doc_scores, dtype=np.float32).transpose()
+                true_vector = subject_set.as_vector(len(self.project.subjects))
                 seq.add_sample(score_vector, true_vector)
 
     def _open_lmdb(self, cached, lmdb_map_size):
@@ -222,11 +225,12 @@ def _open_lmdb(self, cached, lmdb_map_size):
         return lmdb.open(lmdb_path, map_size=lmdb_map_size, writemap=True)
 
     def _fit_model(self, corpus, epochs, lmdb_map_size, n_jobs=1):
-        env = self._open_lmdb(corpus == 'cached', lmdb_map_size)
-        if corpus != 'cached':
+        env = self._open_lmdb(corpus == "cached", lmdb_map_size)
+        if corpus != "cached":
             if corpus.is_empty():
                 raise NotSupportedException(
-                    'Cannot train nn_ensemble project with no documents')
+                    "Cannot train nn_ensemble project with no documents"
+                )
             with env.begin(write=True, buffers=True) as txn:
                 seq = LMDBSequence(txn, batch_size=32)
                 self._corpus_to_vectors(corpus, seq, n_jobs)
@@ -238,14 +242,10 @@ def _fit_model(self, corpus, epochs, lmdb_map_size, n_jobs=1):
             seq = LMDBSequence(txn, batch_size=32)
             self._model.fit(seq, verbose=True, epochs=epochs)
 
-        annif.util.atomic_save(
-            self._model,
-            self.datadir,
-            self.MODEL_FILE)
+        annif.util.atomic_save(self._model, self.datadir, self.MODEL_FILE)
 
     def _learn(self, corpus, params):
         self.initialize()
         self._fit_model(
-            corpus,
-            int(params['learn-epochs']),
-            int(params['lmdb_map_size']))
+            corpus, int(params["learn-epochs"]), int(params["lmdb_map_size"])
+        )
diff --git a/annif/backend/omikuji.py b/annif/backend/omikuji.py
index 15922dcc9..6a6c2505f 100644
--- a/annif/backend/omikuji.py
+++ b/annif/backend/omikuji.py
@@ -5,29 +5,33 @@
 import shutil
 import annif.util
 from annif.suggestion import SubjectSuggestion, ListSuggestionResult
-from annif.exception import NotInitializedException, NotSupportedException, \
-    OperationFailedException
+from annif.exception import (
+    NotInitializedException,
+    NotSupportedException,
+    OperationFailedException,
+)
 from . import backend
 from . import mixins
 
 
 class OmikujiBackend(mixins.TfidfVectorizerMixin, backend.AnnifBackend):
     """Omikuji based backend for Annif"""
+
     name = "omikuji"
 
     # defaults for uninitialized instances
     _model = None
 
-    TRAIN_FILE = 'omikuji-train.txt'
-    MODEL_FILE = 'omikuji-model'
+    TRAIN_FILE = "omikuji-train.txt"
+    MODEL_FILE = "omikuji-model"
 
     DEFAULT_PARAMETERS = {
-        'min_df': 1,
-        'ngram': 1,
-        'cluster_balanced': True,
-        'cluster_k': 2,
-        'max_depth': 20,
-        'collapse_every_n_layers': 0,
+        "min_df": 1,
+        "ngram": 1,
+        "cluster_balanced": True,
+        "cluster_k": 2,
+        "max_depth": 20,
+        "collapse_every_n_layers": 0,
     }
 
     def default_params(self):
@@ -38,76 +42,78 @@ def default_params(self):
     def _initialize_model(self):
         if self._model is None:
             path = os.path.join(self.datadir, self.MODEL_FILE)
-            self.debug('loading model from {}'.format(path))
+            self.debug("loading model from {}".format(path))
             if os.path.exists(path):
                 try:
                     self._model = omikuji.Model.load(path)
                 except RuntimeError:
                     raise OperationFailedException(
                         "Omikuji models trained on Annif versions older than "
-                        "0.56 cannot be loaded. Please retrain your project.")
+                        "0.56 cannot be loaded. Please retrain your project."
+                    )
             else:
                 raise NotInitializedException(
-                    'model {} not found'.format(path),
-                    backend_id=self.backend_id)
+                    "model {} not found".format(path), backend_id=self.backend_id
+                )
 
     def initialize(self, parallel=False):
         self.initialize_vectorizer()
         self._initialize_model()
 
     def _create_train_file(self, veccorpus, corpus):
-        self.info('creating train file')
+        self.info("creating train file")
         path = os.path.join(self.datadir, self.TRAIN_FILE)
-        with open(path, 'w', encoding='utf-8') as trainfile:
+        with open(path, "w", encoding="utf-8") as trainfile:
             # Extreme Classification Repository format header line
             # We don't yet know the number of samples, as some may be skipped
-            print('00000000',
-                  len(self.vectorizer.vocabulary_),
-                  len(self.project.subjects),
-                  file=trainfile)
+            print(
+                "00000000",
+                len(self.vectorizer.vocabulary_),
+                len(self.project.subjects),
+                file=trainfile,
+            )
             n_samples = 0
             for doc, vector in zip(corpus.documents, veccorpus):
-                subject_ids = [str(subject_id)
-                               for subject_id in doc.subject_set]
-                feature_values = ['{}:{}'.format(col, vector[row, col])
-                                  for row, col in zip(*vector.nonzero())]
+                subject_ids = [str(subject_id) for subject_id in doc.subject_set]
+                feature_values = [
+                    "{}:{}".format(col, vector[row, col])
+                    for row, col in zip(*vector.nonzero())
+                ]
                 if not subject_ids or not feature_values:
                     continue  # noqa
-                print(','.join(subject_ids),
-                      ' '.join(feature_values),
-                      file=trainfile)
+                print(",".join(subject_ids), " ".join(feature_values), file=trainfile)
                 n_samples += 1
             # replace the number of samples value at the beginning
             trainfile.seek(0)
-            print('{:08d}'.format(n_samples), end='', file=trainfile)
+            print("{:08d}".format(n_samples), end="", file=trainfile)
 
     def _create_model(self, params, jobs):
         train_path = os.path.join(self.datadir, self.TRAIN_FILE)
         model_path = os.path.join(self.datadir, self.MODEL_FILE)
         hyper_param = omikuji.Model.default_hyper_param()
 
-        hyper_param.cluster_balanced = annif.util.boolean(
-            params['cluster_balanced'])
-        hyper_param.cluster_k = int(params['cluster_k'])
-        hyper_param.max_depth = int(params['max_depth'])
-        hyper_param.collapse_every_n_layers = int(
-            params['collapse_every_n_layers'])
+        hyper_param.cluster_balanced = annif.util.boolean(params["cluster_balanced"])
+        hyper_param.cluster_k = int(params["cluster_k"])
+        hyper_param.max_depth = int(params["max_depth"])
+        hyper_param.collapse_every_n_layers = int(params["collapse_every_n_layers"])
 
-        self._model = omikuji.Model.train_on_data(
-            train_path, hyper_param, jobs or None)
+        self._model = omikuji.Model.train_on_data(train_path, hyper_param, jobs or None)
         if os.path.exists(model_path):
             shutil.rmtree(model_path)
         self._model.save(os.path.join(self.datadir, self.MODEL_FILE))
 
     def _train(self, corpus, params, jobs=0):
-        if corpus != 'cached':
+        if corpus != "cached":
             if corpus.is_empty():
                 raise NotSupportedException(
-                    'Cannot train omikuji project with no documents')
+                    "Cannot train omikuji project with no documents"
+                )
             input = (doc.text for doc in corpus.documents)
-            vecparams = {'min_df': int(params['min_df']),
-                         'tokenizer': self.project.analyzer.tokenize_words,
-                         'ngram_range': (1, int(params['ngram']))}
+            vecparams = {
+                "min_df": int(params["min_df"]),
+                "tokenizer": self.project.analyzer.tokenize_words,
+                "ngram_range": (1, int(params["ngram"])),
+            }
             veccorpus = self.create_vectorizer(input, vecparams)
             self._create_train_file(veccorpus, corpus)
         else:
@@ -115,17 +121,17 @@ def _train(self, corpus, params, jobs=0):
         self._create_model(params, jobs)
 
     def _suggest(self, text, params):
-        self.debug('Suggesting subjects for text "{}..." (len={})'.format(
-            text[:20], len(text)))
+        self.debug(
+            'Suggesting subjects for text "{}..." (len={})'.format(text[:20], len(text))
+        )
         vector = self.vectorizer.transform([text])
         if vector.nnz == 0:  # All zero vector, empty result
             return ListSuggestionResult([])
-        feature_values = [(col, vector[row, col])
-                          for row, col in zip(*vector.nonzero())]
+        feature_values = [
+            (col, vector[row, col]) for row, col in zip(*vector.nonzero())
+        ]
         results = []
-        limit = int(params['limit'])
+        limit = int(params["limit"])
         for subj_id, score in self._model.predict(feature_values, top_k=limit):
-            results.append(SubjectSuggestion(
-                subject_id=subj_id,
-                score=score))
+            results.append(SubjectSuggestion(subject_id=subj_id, score=score))
         return ListSuggestionResult(results)
diff --git a/annif/backend/pav.py b/annif/backend/pav.py
index 99eda9b1c..bea6410b0 100644
--- a/annif/backend/pav.py
+++ b/annif/backend/pav.py
@@ -18,6 +18,7 @@
 
 class PAVBackend(ensemble.BaseEnsembleBackend):
     """PAV ensemble backend that combines results from multiple projects"""
+
     name = "pav"
 
     MODEL_FILE_PREFIX = "pav-model-"
@@ -25,7 +26,7 @@ class PAVBackend(ensemble.BaseEnsembleBackend):
     # defaults for uninitialized instances
     _models = None
 
-    DEFAULT_PARAMETERS = {'min-docs': 10}
+    DEFAULT_PARAMETERS = {"min-docs": 10}
 
     def default_params(self):
         params = backend.AnnifBackend.DEFAULT_PARAMETERS.copy()
@@ -37,17 +38,18 @@ def initialize(self, parallel=False):
         if self._models is not None:
             return  # already initialized
         self._models = {}
-        sources = annif.util.parse_sources(self.params['sources'])
+        sources = annif.util.parse_sources(self.params["sources"])
         for source_project_id, _ in sources:
             model_filename = self.MODEL_FILE_PREFIX + source_project_id
             path = os.path.join(self.datadir, model_filename)
             if os.path.exists(path):
-                self.debug('loading PAV model from {}'.format(path))
+                self.debug("loading PAV model from {}".format(path))
                 self._models[source_project_id] = joblib.load(path)
             else:
                 raise NotInitializedException(
                     "PAV model file '{}' not found".format(path),
-                    backend_id=self.backend_id)
+                    backend_id=self.backend_id,
+                )
 
     def _get_model(self, source_project_id):
         self.initialize()
@@ -63,8 +65,9 @@ def _normalize_hits(self, hits, source_project):
                 score = hit.score
             pav_result.append(
                 annif.suggestion.SubjectSuggestion(
-                    subject_id=hit.subject_id,
-                    score=score))
+                    subject_id=hit.subject_id, score=score
+                )
+            )
         pav_result.sort(key=lambda hit: hit.score, reverse=True)
         return annif.suggestion.ListSuggestionResult(pav_result)
 
@@ -84,22 +87,30 @@ def _suggest_train_corpus(source_project, corpus):
                 row.append(docid)
                 col.append(cid)
             for cid in np.flatnonzero(
-                    doc.subject_set.as_vector(len(source_project.subjects))):
+                doc.subject_set.as_vector(len(source_project.subjects))
+            ):
 
                 trow.append(docid)
                 tcol.append(cid)
             ndocs += 1
-        scores = coo_matrix((data, (row, col)),
-                            shape=(ndocs, len(source_project.subjects)),
-                            dtype=np.float32)
-        true = coo_matrix((np.ones(len(trow), dtype=bool), (trow, tcol)),
-                          shape=(ndocs, len(source_project.subjects)),
-                          dtype=bool)
+        scores = coo_matrix(
+            (data, (row, col)),
+            shape=(ndocs, len(source_project.subjects)),
+            dtype=np.float32,
+        )
+        true = coo_matrix(
+            (np.ones(len(trow), dtype=bool), (trow, tcol)),
+            shape=(ndocs, len(source_project.subjects)),
+            dtype=bool,
+        )
         return csc_matrix(scores), csc_matrix(true)
 
     def _create_pav_model(self, source_project_id, min_docs, corpus):
-        self.info("creating PAV model for source {}, min_docs={}".format(
-            source_project_id, min_docs))
+        self.info(
+            "creating PAV model for source {}, min_docs={}".format(
+                source_project_id, min_docs
+            )
+        )
         source_project = self.project.registry.get_project(source_project_id)
         # suggest subjects for the training corpus
         scores, true = self._suggest_train_corpus(source_project, corpus)
@@ -108,28 +119,27 @@ def _create_pav_model(self, source_project_id, min_docs, corpus):
         for cid in range(len(source_project.subjects)):
             if true[:, cid].sum() < min_docs:
                 continue  # don't create model b/c of too few examples
-            reg = IsotonicRegression(out_of_bounds='clip')
+            reg = IsotonicRegression(out_of_bounds="clip")
             cid_scores = scores[:, cid].toarray().flatten().astype(np.float64)
             reg.fit(cid_scores, true[:, cid].toarray().flatten())
             pav_regressions[cid] = reg
-        self.info("created PAV model for {} concepts".format(
-            len(pav_regressions)))
+        self.info("created PAV model for {} concepts".format(len(pav_regressions)))
         model_filename = self.MODEL_FILE_PREFIX + source_project_id
         annif.util.atomic_save(
-            pav_regressions,
-            self.datadir,
-            model_filename,
-            method=joblib.dump)
+            pav_regressions, self.datadir, model_filename, method=joblib.dump
+        )
 
     def _train(self, corpus, params, jobs=0):
-        if corpus == 'cached':
+        if corpus == "cached":
             raise NotSupportedException(
-                'Training pav project from cached data not supported.')
+                "Training pav project from cached data not supported."
+            )
         if corpus.is_empty():
-            raise NotSupportedException('training backend {} with no documents'
-                                        .format(self.backend_id))
+            raise NotSupportedException(
+                "training backend {} with no documents".format(self.backend_id)
+            )
         self.info("creating PAV models")
-        sources = annif.util.parse_sources(self.params['sources'])
-        min_docs = int(params['min-docs'])
+        sources = annif.util.parse_sources(self.params["sources"])
+        min_docs = int(params["min-docs"])
         for source_project_id, _ in sources:
             self._create_pav_model(source_project_id, min_docs, corpus)
diff --git a/annif/backend/stwfsa.py b/annif/backend/stwfsa.py
index badf344b7..db102cfb6 100644
--- a/annif/backend/stwfsa.py
+++ b/annif/backend/stwfsa.py
@@ -6,20 +6,18 @@
 from annif.util import atomic_save, boolean
 
 
-_KEY_CONCEPT_TYPE_URI = 'concept_type_uri'
-_KEY_SUBTHESAURUS_TYPE_URI = 'sub_thesaurus_type_uri'
-_KEY_THESAURUS_RELATION_TYPE_URI = 'thesaurus_relation_type_uri'
-_KEY_THESAURUS_RELATION_IS_SPECIALISATION = (
-    'thesaurus_relation_is_specialisation')
-_KEY_REMOVE_DEPRECATED = 'remove_deprecated'
-_KEY_HANDLE_TITLE_CASE = 'handle_title_case'
-_KEY_EXTRACT_UPPER_CASE_FROM_BRACES = 'extract_upper_case_from_braces'
-_KEY_EXTRACT_ANY_CASE_FROM_BRACES = 'extract_any_case_from_braces'
-_KEY_EXPAND_AMPERSAND_WITH_SPACES = 'expand_ampersand_with_spaces'
-_KEY_EXPAND_ABBREVIATION_WITH_PUNCTUATION = (
-    'expand_abbreviation_with_punctuation')
-_KEY_SIMPLE_ENGLISH_PLURAL_RULES = 'simple_english_plural_rules'
-_KEY_USE_TXT_VEC = 'use_txt_vec'
+_KEY_CONCEPT_TYPE_URI = "concept_type_uri"
+_KEY_SUBTHESAURUS_TYPE_URI = "sub_thesaurus_type_uri"
+_KEY_THESAURUS_RELATION_TYPE_URI = "thesaurus_relation_type_uri"
+_KEY_THESAURUS_RELATION_IS_SPECIALISATION = "thesaurus_relation_is_specialisation"
+_KEY_REMOVE_DEPRECATED = "remove_deprecated"
+_KEY_HANDLE_TITLE_CASE = "handle_title_case"
+_KEY_EXTRACT_UPPER_CASE_FROM_BRACES = "extract_upper_case_from_braces"
+_KEY_EXTRACT_ANY_CASE_FROM_BRACES = "extract_any_case_from_braces"
+_KEY_EXPAND_AMPERSAND_WITH_SPACES = "expand_ampersand_with_spaces"
+_KEY_EXPAND_ABBREVIATION_WITH_PUNCTUATION = "expand_abbreviation_with_punctuation"
+_KEY_SIMPLE_ENGLISH_PLURAL_RULES = "simple_english_plural_rules"
+_KEY_USE_TXT_VEC = "use_txt_vec"
 
 
 class StwfsaBackend(backend.AnnifBackend):
@@ -42,11 +40,9 @@ class StwfsaBackend(backend.AnnifBackend):
     }
 
     DEFAULT_PARAMETERS = {
-        _KEY_CONCEPT_TYPE_URI: 'http://www.w3.org/2004/02/skos/core#Concept',
-        _KEY_SUBTHESAURUS_TYPE_URI:
-            'http://www.w3.org/2004/02/skos/core#Collection',
-        _KEY_THESAURUS_RELATION_TYPE_URI:
-            'http://www.w3.org/2004/02/skos/core#member',
+        _KEY_CONCEPT_TYPE_URI: "http://www.w3.org/2004/02/skos/core#Concept",
+        _KEY_SUBTHESAURUS_TYPE_URI: "http://www.w3.org/2004/02/skos/core#Collection",
+        _KEY_THESAURUS_RELATION_TYPE_URI: "http://www.w3.org/2004/02/skos/core#member",
         _KEY_THESAURUS_RELATION_IS_SPECIALISATION: True,
         _KEY_REMOVE_DEPRECATED: True,
         _KEY_HANDLE_TITLE_CASE: True,
@@ -55,69 +51,76 @@ class StwfsaBackend(backend.AnnifBackend):
         _KEY_EXPAND_AMPERSAND_WITH_SPACES: True,
         _KEY_EXPAND_ABBREVIATION_WITH_PUNCTUATION: True,
         _KEY_SIMPLE_ENGLISH_PLURAL_RULES: False,
-        _KEY_USE_TXT_VEC: False
+        _KEY_USE_TXT_VEC: False,
     }
 
-    MODEL_FILE = 'stwfsa_predictor.zip'
+    MODEL_FILE = "stwfsa_predictor.zip"
 
     _model = None
 
     def initialize(self, parallel=False):
         if self._model is None:
             path = os.path.join(self.datadir, self.MODEL_FILE)
-            self.debug(f'Loading STWFSA model from {path}.')
+            self.debug(f"Loading STWFSA model from {path}.")
             if os.path.exists(path):
                 self._model = StwfsapyPredictor.load(path)
-                self.debug('Loaded model.')
+                self.debug("Loaded model.")
             else:
                 raise NotInitializedException(
-                    f'Model not found at {path}',
-                    backend_id=self.backend_id)
+                    f"Model not found at {path}", backend_id=self.backend_id
+                )
 
     def _load_data(self, corpus):
-        if corpus == 'cached':
+        if corpus == "cached":
             raise NotSupportedException(
-                'Training stwfsa project from cached data not supported.')
+                "Training stwfsa project from cached data not supported."
+            )
         if corpus.is_empty():
             raise NotSupportedException(
-                'Cannot train stwfsa project with no documents.')
+                "Cannot train stwfsa project with no documents."
+            )
         self.debug("Transforming training data.")
         X = []
         y = []
         for doc in corpus.documents:
             X.append(doc.text)
-            y.append([self.project.subjects[subject_id].uri
-                      for subject_id in doc.subject_set])
+            y.append(
+                [
+                    self.project.subjects[subject_id].uri
+                    for subject_id in doc.subject_set
+                ]
+            )
         return X, y
 
     def _train(self, corpus, params, jobs=0):
         X, y = self._load_data(corpus)
         new_params = {
-                key: self.STWFSA_PARAMETERS[key](val)
-                for key, val
-                in params.items()
-                if key in self.STWFSA_PARAMETERS
-            }
+            key: self.STWFSA_PARAMETERS[key](val)
+            for key, val in params.items()
+            if key in self.STWFSA_PARAMETERS
+        }
         p = StwfsapyPredictor(
             graph=self.project.vocab.as_graph(),
-            langs=frozenset([params['language']]),
-            **new_params)
+            langs=frozenset([params["language"]]),
+            **new_params,
+        )
         p.fit(X, y)
         self._model = p
         atomic_save(
             p,
             self.datadir,
             self.MODEL_FILE,
-            lambda model, store_path: model.store(store_path))
+            lambda model, store_path: model.store(store_path),
+        )
 
     def _suggest(self, text, params):
-        self.debug(
-            f'Suggesting subjects for text "{text[:20]}..." (len={len(text)})')
+        self.debug(f'Suggesting subjects for text "{text[:20]}..." (len={len(text)})')
         result = self._model.suggest_proba([text])[0]
         suggestions = []
         for uri, score in result:
             subject_id = self.project.subjects.by_uri(uri)
             if subject_id is not None:
-                suggestions.append(SubjectSuggestion(subject_id=subject_id,
-                                                     score=score))
+                suggestions.append(
+                    SubjectSuggestion(subject_id=subject_id, score=score)
+                )
         return ListSuggestionResult(suggestions)
diff --git a/annif/backend/svc.py b/annif/backend/svc.py
index 4383d5b1f..b8358f229 100644
--- a/annif/backend/svc.py
+++ b/annif/backend/svc.py
@@ -14,17 +14,15 @@
 
 class SVCBackend(mixins.TfidfVectorizerMixin, backend.AnnifBackend):
     """Support vector classifier backend for Annif"""
+
     name = "svc"
 
     # defaults for uninitialized instances
     _model = None
 
-    MODEL_FILE = 'svc-model.gz'
+    MODEL_FILE = "svc-model.gz"
 
-    DEFAULT_PARAMETERS = {
-        'min_df': 1,
-        'ngram': 1
-    }
+    DEFAULT_PARAMETERS = {"min_df": 1, "ngram": 1}
 
     def default_params(self):
         params = backend.AnnifBackend.DEFAULT_PARAMETERS.copy()
@@ -34,13 +32,13 @@ def default_params(self):
     def _initialize_model(self):
         if self._model is None:
             path = os.path.join(self.datadir, self.MODEL_FILE)
-            self.debug('loading model from {}'.format(path))
+            self.debug("loading model from {}".format(path))
             if os.path.exists(path):
                 self._model = joblib.load(path)
             else:
                 raise NotInitializedException(
-                    'model {} not found'.format(path),
-                    backend_id=self.backend_id)
+                    "model {} not found".format(path), backend_id=self.backend_id
+                )
 
     def initialize(self, parallel=False):
         self.initialize_vectorizer()
@@ -52,8 +50,9 @@ def _corpus_to_texts_and_classes(self, corpus):
         for doc in corpus.documents:
             if len(doc.subject_set) > 1:
                 self.warning(
-                    'training on a document with multiple subjects is not ' +
-                    'supported by SVC; selecting one random subject.')
+                    "training on a document with multiple subjects is not "
+                    + "supported by SVC; selecting one random subject."
+                )
             elif not doc.subject_set:
                 continue  # skip documents with no subjects
             texts.append(doc.text)
@@ -61,42 +60,44 @@ def _corpus_to_texts_and_classes(self, corpus):
         return texts, classes
 
     def _train_classifier(self, veccorpus, classes):
-        self.info('creating classifier')
+        self.info("creating classifier")
         self._model = LinearSVC()
         self._model.fit(veccorpus, classes)
-        annif.util.atomic_save(self._model,
-                               self.datadir,
-                               self.MODEL_FILE,
-                               method=joblib.dump)
+        annif.util.atomic_save(
+            self._model, self.datadir, self.MODEL_FILE, method=joblib.dump
+        )
 
     def _train(self, corpus, params, jobs=0):
-        if corpus == 'cached':
+        if corpus == "cached":
             raise NotSupportedException(
-                'SVC backend does not support reuse of cached training data.')
+                "SVC backend does not support reuse of cached training data."
+            )
         if corpus.is_empty():
-            raise NotSupportedException(
-                'Cannot train SVC project with no documents')
+            raise NotSupportedException("Cannot train SVC project with no documents")
         texts, classes = self._corpus_to_texts_and_classes(corpus)
-        vecparams = {'min_df': int(params['min_df']),
-                     'tokenizer': self.project.analyzer.tokenize_words,
-                     'ngram_range': (1, int(params['ngram']))}
+        vecparams = {
+            "min_df": int(params["min_df"]),
+            "tokenizer": self.project.analyzer.tokenize_words,
+            "ngram_range": (1, int(params["ngram"])),
+        }
         veccorpus = self.create_vectorizer(texts, vecparams)
         self._train_classifier(veccorpus, classes)
 
     def _scores_to_suggestions(self, scores, params):
         results = []
-        limit = int(params['limit'])
+        limit = int(params["limit"])
         for class_id in np.argsort(scores)[::-1][:limit]:
             subject_id = self._model.classes_[class_id]
             if subject_id is not None:
-                results.append(SubjectSuggestion(
-                    subject_id=subject_id,
-                    score=scores[class_id]))
+                results.append(
+                    SubjectSuggestion(subject_id=subject_id, score=scores[class_id])
+                )
         return ListSuggestionResult(results)
 
     def _suggest(self, text, params):
-        self.debug('Suggesting subjects for text "{}..." (len={})'.format(
-            text[:20], len(text)))
+        self.debug(
+            'Suggesting subjects for text "{}..." (len={})'.format(text[:20], len(text))
+        )
         vector = self.vectorizer.transform([text])
         if vector.nnz == 0:  # All zero vector, empty result
             return ListSuggestionResult([])
diff --git a/annif/backend/tfidf.py b/annif/backend/tfidf.py
index 738296ea6..fad58d905 100644
--- a/annif/backend/tfidf.py
+++ b/annif/backend/tfidf.py
@@ -18,18 +18,18 @@ class SubjectBuffer:
     BUFFER_SIZE = 100
 
     def __init__(self, tempdir, subject_id):
-        filename = '{:08d}.txt'.format(subject_id)
+        filename = "{:08d}.txt".format(subject_id)
         self._path = os.path.join(tempdir, filename)
         self._buffer = []
         self._created = False
 
     def flush(self):
         if self._created:
-            mode = 'a'
+            mode = "a"
         else:
-            mode = 'w'
+            mode = "w"
 
-        with open(self._path, mode, encoding='utf-8') as subjfile:
+        with open(self._path, mode, encoding="utf-8") as subjfile:
             for text in self._buffer:
                 print(text, file=subjfile)
 
@@ -46,25 +46,25 @@ def read(self):
             # file was never created - we can simply return the buffer content
             return "\n".join(self._buffer)
         else:
-            with open(self._path, 'r', encoding='utf-8') as subjfile:
+            with open(self._path, "r", encoding="utf-8") as subjfile:
                 return subjfile.read() + "\n" + "\n".join(self._buffer)
 
 
 class TFIDFBackend(mixins.TfidfVectorizerMixin, backend.AnnifBackend):
     """TF-IDF vector space similarity based backend for Annif"""
+
     name = "tfidf"
 
     # defaults for uninitialized instances
     _index = None
 
-    INDEX_FILE = 'tfidf-index'
+    INDEX_FILE = "tfidf-index"
 
     def _generate_subjects_from_documents(self, corpus):
         with tempfile.TemporaryDirectory() as tempdir:
             subject_buffer = {}
             for subject_id in range(len(self.project.subjects)):
-                subject_buffer[subject_id] = SubjectBuffer(tempdir,
-                                                           subject_id)
+                subject_buffer[subject_id] = SubjectBuffer(tempdir, subject_id)
 
             for doc in corpus.documents:
                 tokens = self.project.analyzer.tokenize_words(doc.text)
@@ -77,47 +77,45 @@ def _generate_subjects_from_documents(self, corpus):
     def _initialize_index(self):
         if self._index is None:
             path = os.path.join(self.datadir, self.INDEX_FILE)
-            self.debug('loading similarity index from {}'.format(path))
+            self.debug("loading similarity index from {}".format(path))
             if os.path.exists(path):
-                self._index = gensim.similarities.SparseMatrixSimilarity.load(
-                    path)
+                self._index = gensim.similarities.SparseMatrixSimilarity.load(path)
             else:
                 raise NotInitializedException(
-                    'similarity index {} not found'.format(path),
-                    backend_id=self.backend_id)
+                    "similarity index {} not found".format(path),
+                    backend_id=self.backend_id,
+                )
 
     def initialize(self, parallel=False):
         self.initialize_vectorizer()
         self._initialize_index()
 
     def _create_index(self, veccorpus):
-        self.info('creating similarity index')
+        self.info("creating similarity index")
         gscorpus = Sparse2Corpus(veccorpus, documents_columns=False)
         self._index = gensim.similarities.SparseMatrixSimilarity(
-            gscorpus, num_features=len(self.vectorizer.vocabulary_))
-        annif.util.atomic_save(
-            self._index,
-            self.datadir,
-            self.INDEX_FILE)
+            gscorpus, num_features=len(self.vectorizer.vocabulary_)
+        )
+        annif.util.atomic_save(self._index, self.datadir, self.INDEX_FILE)
 
     def _train(self, corpus, params, jobs=0):
-        if corpus == 'cached':
+        if corpus == "cached":
             raise NotSupportedException(
-                'Training tfidf project from cached data not supported.')
+                "Training tfidf project from cached data not supported."
+            )
         if corpus.is_empty():
-            raise NotSupportedException(
-                'Cannot train tfidf project with no documents')
-        self.info('transforming subject corpus')
+            raise NotSupportedException("Cannot train tfidf project with no documents")
+        self.info("transforming subject corpus")
         subjects = self._generate_subjects_from_documents(corpus)
         veccorpus = self.create_vectorizer(subjects)
         self._create_index(veccorpus)
 
     def _suggest(self, text, params):
-        self.debug('Suggesting subjects for text "{}..." (len={})'.format(
-            text[:20], len(text)))
+        self.debug(
+            'Suggesting subjects for text "{}..." (len={})'.format(text[:20], len(text))
+        )
         tokens = self.project.analyzer.tokenize_words(text)
         vectors = self.vectorizer.transform([" ".join(tokens)])
         docsim = self._index[vectors[0]]
         fullresult = VectorSuggestionResult(docsim)
-        return fullresult.filter(self.project.subjects,
-                                 limit=int(params['limit']))
+        return fullresult.filter(self.project.subjects, limit=int(params["limit"]))
diff --git a/annif/backend/yake.py b/annif/backend/yake.py
index 747ddf2e5..8773d4964 100644
--- a/annif/backend/yake.py
+++ b/annif/backend/yake.py
@@ -16,22 +16,23 @@
 
 class YakeBackend(backend.AnnifBackend):
     """Yake based backend for Annif"""
+
     name = "yake"
 
     # defaults for uninitialized instances
     _index = None
     _graph = None
-    INDEX_FILE = 'yake-index'
+    INDEX_FILE = "yake-index"
 
     DEFAULT_PARAMETERS = {
-        'max_ngram_size': 4,
-        'deduplication_threshold': 0.9,
-        'deduplication_algo': 'levs',
-        'window_size': 1,
-        'num_keywords': 100,
-        'features': None,
-        'label_types': ['prefLabel', 'altLabel'],
-        'remove_parentheses': False
+        "max_ngram_size": 4,
+        "deduplication_threshold": 0.9,
+        "deduplication_algo": "levs",
+        "window_size": 1,
+        "num_keywords": 100,
+        "features": None,
+        "label_types": ["prefLabel", "altLabel"],
+        "remove_parentheses": False,
     }
 
     def default_params(self):
@@ -45,19 +46,19 @@ def is_trained(self):
 
     @property
     def label_types(self):
-        if type(self.params['label_types']) == str:  # Label types set by user
-            label_types = [lt.strip() for lt
-                           in self.params['label_types'].split(',')]
+        if type(self.params["label_types"]) == str:  # Label types set by user
+            label_types = [lt.strip() for lt in self.params["label_types"].split(",")]
             self._validate_label_types(label_types)
         else:
-            label_types = self.params['label_types']  # The defaults
+            label_types = self.params["label_types"]  # The defaults
         return [getattr(SKOS, lt) for lt in label_types]
 
     def _validate_label_types(self, label_types):
         for lt in label_types:
-            if lt not in ('prefLabel', 'altLabel', 'hiddenLabel'):
+            if lt not in ("prefLabel", "altLabel", "hiddenLabel"):
                 raise ConfigurationException(
-                    f'invalid label type {lt}', backend_id=self.backend_id)
+                    f"invalid label type {lt}", backend_id=self.backend_id
+                )
 
     def initialize(self, parallel=False):
         self._initialize_index()
@@ -67,69 +68,65 @@ def _initialize_index(self):
             path = os.path.join(self.datadir, self.INDEX_FILE)
             if os.path.exists(path):
                 self._index = joblib.load(path)
-                self.debug(
-                    f'Loaded index from {path} with {len(self._index)} labels')
+                self.debug(f"Loaded index from {path} with {len(self._index)} labels")
             else:
-                self.info('Creating index')
+                self.info("Creating index")
                 self._index = self._create_index()
                 self._save_index(path)
-                self.info(f'Created index with {len(self._index)} labels')
+                self.info(f"Created index with {len(self._index)} labels")
 
     def _save_index(self, path):
         annif.util.atomic_save(
-            self._index,
-            self.datadir,
-            self.INDEX_FILE,
-            method=joblib.dump)
+            self._index, self.datadir, self.INDEX_FILE, method=joblib.dump
+        )
 
     def _create_index(self):
         index = defaultdict(set)
         skos_vocab = self.project.vocab.skos
         for concept in skos_vocab.concepts:
             uri = str(concept)
-            labels_by_lang = skos_vocab.get_concept_labels(concept,
-                                                           self.label_types)
-            for label in labels_by_lang[self.params['language']]:
+            labels_by_lang = skos_vocab.get_concept_labels(concept, self.label_types)
+            for label in labels_by_lang[self.params["language"]]:
                 label = self._normalize_label(label)
                 index[label].add(uri)
-        index.pop('', None)  # Remove possible empty string entry
+        index.pop("", None)  # Remove possible empty string entry
         return dict(index)
 
     def _normalize_label(self, label):
         label = str(label)
-        if annif.util.boolean(self.params['remove_parentheses']):
-            label = re.sub(r' \(.*\)', '', label)
+        if annif.util.boolean(self.params["remove_parentheses"]):
+            label = re.sub(r" \(.*\)", "", label)
         normalized_label = self._normalize_phrase(label)
         return self._sort_phrase(normalized_label)
 
     def _normalize_phrase(self, phrase):
-        return ' '.join(self.project.analyzer.tokenize_words(phrase,
-                                                             filter=False))
+        return " ".join(self.project.analyzer.tokenize_words(phrase, filter=False))
 
     def _sort_phrase(self, phrase):
         words = phrase.split()
-        return ' '.join(sorted(words))
+        return " ".join(sorted(words))
 
     def _suggest(self, text, params):
-        self.debug(
-            f'Suggesting subjects for text "{text[:20]}..." (len={len(text)})')
-        limit = int(params['limit'])
+        self.debug(f'Suggesting subjects for text "{text[:20]}..." (len={len(text)})')
+        limit = int(params["limit"])
 
         self._kw_extractor = yake.KeywordExtractor(
-            lan=params['language'],
-            n=int(params['max_ngram_size']),
-            dedupLim=float(params['deduplication_threshold']),
-            dedupFunc=params['deduplication_algo'],
-            windowsSize=int(params['window_size']),
-            top=int(params['num_keywords']),
-            features=self.params['features'])
+            lan=params["language"],
+            n=int(params["max_ngram_size"]),
+            dedupLim=float(params["deduplication_threshold"]),
+            dedupFunc=params["deduplication_algo"],
+            windowsSize=int(params["window_size"]),
+            top=int(params["num_keywords"]),
+            features=self.params["features"],
+        )
         keyphrases = self._kw_extractor.extract_keywords(text)
         suggestions = self._keyphrases2suggestions(keyphrases)
 
-        subject_suggestions = [SubjectSuggestion(
-                subject_id=self.project.subjects.by_uri(uri),
-                score=score)
-                for uri, score in suggestions[:limit] if score > 0.0]
+        subject_suggestions = [
+            SubjectSuggestion(subject_id=self.project.subjects.by_uri(uri), score=score)
+            for uri, score in suggestions[:limit]
+            if score > 0.0
+        ]
         return ListSuggestionResult(subject_suggestions)
 
     def _keyphrases2suggestions(self, keyphrases):
@@ -138,15 +135,20 @@ def _keyphrases2suggestions(self, keyphrases):
         for kp, score in keyphrases:
             uris = self._keyphrase2uris(kp)
             for uri in uris:
-                suggestions.append(
-                    (uri, self._transform_score(score)))
+                suggestions.append((uri, self._transform_score(score)))
             if not uris:
                 not_matched.append((kp, self._transform_score(score)))
         # Remove duplicate uris, conflating the scores
         suggestions = self._combine_suggestions(suggestions)
-        self.debug('Keyphrases not matched:\n' + '\t'.join(
-            [kp[0] + ' ' + str(kp[1]) for kp
-             in sorted(not_matched, reverse=True, key=lambda kp: kp[1])]))
+        self.debug(
+            "Keyphrases not matched:\n"
+            + "\t".join(
+                [
+                    kp[0] + " " + str(kp[1])
+                    for kp in sorted(not_matched, reverse=True, key=lambda kp: kp[1])
+                ]
+            )
+        )
         return suggestions
 
     def _keyphrase2uris(self, keyphrase):
@@ -165,17 +167,15 @@ def _combine_suggestions(self, suggestions):
                 combined_suggestions[uri] = score
             else:
                 old_score = combined_suggestions[uri]
-                combined_suggestions[uri] = self._combine_scores(
-                    score, old_score)
+                combined_suggestions[uri] = self._combine_scores(score, old_score)
         return list(combined_suggestions.items())
 
     def _combine_scores(self, score1, score2):
         # The result is never smaller than the greater input
-        score1 = score1/2 + 0.5
-        score2 = score2/2 + 0.5
-        confl = score1 * score2 / (score1 * score2 + (1-score1) * (1-score2))
-        return (confl-0.5) * 2
+        score1 = score1 / 2 + 0.5
+        score2 = score2 / 2 + 0.5
+        confl = score1 * score2 / (score1 * score2 + (1 - score1) * (1 - score2))
+        return (confl - 0.5) * 2
 
     def _train(self, corpus, params, jobs=0):
-        raise NotSupportedException(
-            'Training yake backend is not possible.')
+        raise NotSupportedException("Training yake backend is not possible.")
diff --git a/annif/cli.py b/annif/cli.py
index bcc56b0fa..f39826309 100644
--- a/annif/cli.py
+++ b/annif/cli.py
@@ -26,19 +26,16 @@
 click_log.basic_config(logger)
 
 cli = FlaskGroup(create_app=annif.create_app, add_version_option=False)
-cli = click.version_option(message='%(version)s')(cli)
+cli = click.version_option(message="%(version)s")(cli)
 
 
 def get_project(project_id):
     """
     Helper function to get a project by ID and bail out if it doesn't exist"""
     try:
-        return annif.registry.get_project(project_id,
-                                          min_access=Access.private)
+        return annif.registry.get_project(project_id, min_access=Access.private)
     except ValueError:
-        click.echo(
-            "No projects found with id \'{0}\'.".format(project_id),
-            err=True)
+        click.echo("No projects found with id '{0}'.".format(project_id), err=True)
         sys.exit(1)
 
 
@@ -47,12 +44,9 @@ def get_vocab(vocab_id):
     Helper function to get a vocabulary by ID and bail out if it doesn't
     exist"""
     try:
-        return annif.registry.get_vocab(vocab_id,
-                                        min_access=Access.private)
+        return annif.registry.get_vocab(vocab_id, min_access=Access.private)
     except ValueError:
-        click.echo(
-            f"No vocabularies found with the id '{vocab_id}'.",
-            err=True)
+        click.echo(f"No vocabularies found with the id '{vocab_id}'.", err=True)
         sys.exit(1)
 
 
@@ -66,13 +60,13 @@ def open_documents(paths, subject_index, vocab_lang, docs_limit):
     def open_doc_path(path, subject_index):
         """open a single path and return it as a DocumentCorpus"""
         if os.path.isdir(path):
-            return annif.corpus.DocumentDirectory(path, subject_index,
-                                                  vocab_lang,
-                                                  require_subjects=True)
+            return annif.corpus.DocumentDirectory(
+                path, subject_index, vocab_lang, require_subjects=True
+            )
         return annif.corpus.DocumentFile(path, subject_index)
 
     if len(paths) == 0:
-        logger.warning('Reading empty file')
+        logger.warning("Reading empty file")
         docs = open_doc_path(os.path.devnull, subject_index)
     elif len(paths) == 1:
         docs = open_doc_path(paths[0], subject_index)
@@ -89,19 +83,19 @@ def parse_backend_params(backend_param, project):
     option into a nested dict structure"""
     backend_params = collections.defaultdict(dict)
     for beparam in backend_param:
-        backend, param = beparam.split('.', 1)
-        key, val = param.split('=', 1)
+        backend, param = beparam.split(".", 1)
+        key, val = param.split("=", 1)
         validate_backend_params(backend, beparam, project)
         backend_params[backend][key] = val
     return backend_params
 
 
 def validate_backend_params(backend, beparam, project):
-    if backend != project.config['backend']:
+    if backend != project.config["backend"]:
         raise ConfigurationException(
             'The backend {} in CLI option "-b {}" not matching the project'
-            ' backend {}.'
-            .format(backend, beparam, project.config['backend']))
+            " backend {}.".format(backend, beparam, project.config["backend"])
+        )
 
 
 BATCH_MAX_LIMIT = 15
@@ -109,6 +103,7 @@ def validate_backend_params(backend, beparam, project):
 
 def generate_filter_batches(subjects):
     import annif.eval
+
     filter_batches = collections.OrderedDict()
     for limit in range(1, BATCH_MAX_LIMIT + 1):
         for threshold in [i * 0.05 for i in range(20)]:
@@ -122,31 +117,37 @@ def set_project_config_file_path(ctx, param, value):
     """Override the default path or the path given in env by CLI option"""
     with ctx.ensure_object(ScriptInfo).load_app().app_context():
         if value:
-            current_app.config['PROJECTS_CONFIG_PATH'] = value
+            current_app.config["PROJECTS_CONFIG_PATH"] = value
 
 
 def common_options(f):
     """Decorator to add common options for all CLI commands"""
     f = click.option(
-        '-p', '--projects',
-        help='Set path to project configuration file or directory',
+        "-p",
+        "--projects",
+        help="Set path to project configuration file or directory",
         type=click.Path(dir_okay=True, exists=True),
-        callback=set_project_config_file_path, expose_value=False,
-        is_eager=True)(f)
+        callback=set_project_config_file_path,
+        expose_value=False,
+        is_eager=True,
+    )(f)
     return click_log.simple_verbosity_option(logger)(f)
 
 
 def backend_param_option(f):
     """Decorator to add an option for CLI commands to override BE parameters"""
     return click.option(
-        '--backend-param', '-b', multiple=True,
-        help='Override backend parameter of the config file. ' +
-        'Syntax: `-b <backend>.<parameter>=<value>`.')(f)
+        "--backend-param",
+        "-b",
+        multiple=True,
+        help="Override backend parameter of the config file. "
+        + "Syntax: `-b <backend>.<parameter>=<value>`.",
+    )(f)
 
 
-@cli.command('list-projects')
+@cli.command("list-projects")
 @common_options
-@click_log.simple_verbosity_option(logger, default='ERROR')
+@click_log.simple_verbosity_option(logger, default="ERROR")
 def run_list_projects():
     """
     List available projects.
@@ -159,18 +160,19 @@ def run_list_projects():
     """
 
     template = "{0: <25}{1: <45}{2: <10}{3: <7}"
-    header = template.format(
-        "Project ID", "Project Name", "Language", "Trained")
+    header = template.format("Project ID", "Project Name", "Language", "Trained")
     click.echo(header)
     click.echo("-" * len(header))
-    for proj in annif.registry.get_projects(
-            min_access=Access.private).values():
-        click.echo(template.format(
-            proj.project_id, proj.name, proj.language, str(proj.is_trained)))
+    for proj in annif.registry.get_projects(min_access=Access.private).values():
+        click.echo(
+            template.format(
+                proj.project_id, proj.name, proj.language, str(proj.is_trained)
+            )
+        )
 
 
-@cli.command('show-project')
-@click.argument('project_id')
+@cli.command("show-project")
+@click.argument("project_id")
 @common_options
 def run_show_project(project_id):
     """
@@ -178,18 +180,18 @@ def run_show_project(project_id):
     """
 
     proj = get_project(project_id)
-    click.echo(f'Project ID:        {proj.project_id}')
-    click.echo(f'Project Name:      {proj.name}')
-    click.echo(f'Language:          {proj.language}')
-    click.echo(f'Vocabulary:        {proj.vocab.vocab_id}')
-    click.echo(f'Vocab language:    {proj.vocab_lang}')
-    click.echo(f'Access:            {proj.access.name}')
-    click.echo(f'Trained:           {proj.is_trained}')
-    click.echo(f'Modification time: {proj.modification_time}')
-
-
-@cli.command('clear')
-@click.argument('project_id')
+    click.echo(f"Project ID:        {proj.project_id}")
+    click.echo(f"Project Name:      {proj.name}")
+    click.echo(f"Language:          {proj.language}")
+    click.echo(f"Vocabulary:        {proj.vocab.vocab_id}")
+    click.echo(f"Vocab language:    {proj.vocab_lang}")
+    click.echo(f"Access:            {proj.access.name}")
+    click.echo(f"Trained:           {proj.is_trained}")
+    click.echo(f"Modification time: {proj.modification_time}")
+
+
+@cli.command("clear")
+@click.argument("project_id")
 @common_options
 def run_clear_project(project_id):
     """
@@ -199,40 +201,41 @@ def run_clear_project(project_id):
     proj.remove_model_data()
 
 
-@cli.command('list-vocabs')
+@cli.command("list-vocabs")
 @common_options
-@click_log.simple_verbosity_option(logger, default='ERROR')
+@click_log.simple_verbosity_option(logger, default="ERROR")
 def run_list_vocabs():
     """
     List available vocabularies.
     """
 
     template = "{0: <20}{1: <20}{2: >10}  {3: <6}"
-    header = template.format(
-        "Vocabulary ID", "Languages", "Size", "Loaded")
+    header = template.format("Vocabulary ID", "Languages", "Size", "Loaded")
     click.echo(header)
     click.echo("-" * len(header))
-    for vocab in annif.registry.get_vocabs(
-            min_access=Access.private).values():
+    for vocab in annif.registry.get_vocabs(min_access=Access.private).values():
         try:
-            languages = ','.join(sorted(vocab.languages))
+            languages = ",".join(sorted(vocab.languages))
             size = len(vocab)
             loaded = True
         except NotInitializedException:
-            languages = '-'
-            size = '-'
+            languages = "-"
+            size = "-"
             loaded = False
-        click.echo(template.format(
-            vocab.vocab_id, languages, size, str(loaded)))
+        click.echo(template.format(vocab.vocab_id, languages, size, str(loaded)))
 
 
-@cli.command('load-vocab')
-@click.argument('vocab_id')
-@click.argument('subjectfile', type=click.Path(exists=True, dir_okay=False))
-@click.option('--language', '-L', help='Language of subject file')
-@click.option('--force', '-f', default=False, is_flag=True,
-              help='Replace existing vocabulary completely ' +
-                   'instead of updating it')
+@cli.command("load-vocab")
+@click.argument("vocab_id")
+@click.argument("subjectfile", type=click.Path(exists=True, dir_okay=False))
+@click.option("--language", "-L", help="Language of subject file")
+@click.option(
+    "--force",
+    "-f",
+    default=False,
+    is_flag=True,
+    help="Replace existing vocabulary completely " + "instead of updating it",
+)
 @common_options
 def run_load_vocab(vocab_id, language, force, subjectfile):
     """
@@ -250,26 +253,39 @@ def run_load_vocab(vocab_id, language, force, subjectfile):
     else:
         # probably a TSV file - we need to know its language
         if not language:
-            click.echo("Please use --language option to set the language of " +
-                       "a TSV vocabulary.", err=True)
+            click.echo(
+                "Please use --language option to set the language of "
+                + "a TSV vocabulary.",
+                err=True,
+            )
             sys.exit(1)
         click.echo(f"Loading vocabulary from TSV file {subjectfile}...")
         subjects = annif.corpus.SubjectFileTSV(subjectfile, language)
     vocab.load_vocabulary(subjects, force=force)
 
 
-@cli.command('train')
-@click.argument('project_id')
-@click.argument('paths', type=click.Path(exists=True), nargs=-1)
-@click.option('--cached/--no-cached', '-c/-C', default=False,
-              help='Reuse preprocessed training data from previous run')
-@click.option('--docs-limit', '-d', default=None,
-              type=click.IntRange(0, None),
-              help='Maximum number of documents to use')
-@click.option('--jobs',
-              '-j',
-              default=0,
-              help='Number of parallel jobs (0 means choose automatically)')
+@cli.command("train")
+@click.argument("project_id")
+@click.argument("paths", type=click.Path(exists=True), nargs=-1)
+@click.option(
+    "--cached/--no-cached",
+    "-c/-C",
+    default=False,
+    help="Reuse preprocessed training data from previous run",
+)
+@click.option(
+    "--docs-limit",
+    "-d",
+    default=None,
+    type=click.IntRange(0, None),
+    help="Maximum number of documents to use",
+)
+@click.option(
+    "--jobs",
+    "-j",
+    default=0,
+    help="Number of parallel jobs (0 means choose automatically)",
+)
 @backend_param_option
 @common_options
 def run_train(project_id, paths, cached, docs_limit, jobs, backend_param):
@@ -288,20 +304,24 @@ def run_train(project_id, paths, cached, docs_limit, jobs, backend_param):
     if cached:
         if len(paths) > 0:
             raise click.UsageError(
-                "Corpus paths cannot be given when using --cached option.")
-        documents = 'cached'
+                "Corpus paths cannot be given when using --cached option."
+            )
+        documents = "cached"
     else:
-        documents = open_documents(paths, proj.subjects,
-                                   proj.vocab_lang, docs_limit)
+        documents = open_documents(paths, proj.subjects, proj.vocab_lang, docs_limit)
     proj.train(documents, backend_params, jobs)
 
 
-@cli.command('learn')
-@click.argument('project_id')
-@click.argument('paths', type=click.Path(exists=True), nargs=-1)
-@click.option('--docs-limit', '-d', default=None,
-              type=click.IntRange(0, None),
-              help='Maximum number of documents to use')
+@cli.command("learn")
+@click.argument("project_id")
+@click.argument("paths", type=click.Path(exists=True), nargs=-1)
+@click.option(
+    "--docs-limit",
+    "-d",
+    default=None,
+    type=click.IntRange(0, None),
+    help="Maximum number of documents to use",
+)
 @backend_param_option
 @common_options
 def run_learn(project_id, paths, docs_limit, backend_param):
@@ -314,16 +334,15 @@ def run_learn(project_id, paths, docs_limit, backend_param):
     """
     proj = get_project(project_id)
     backend_params = parse_backend_params(backend_param, proj)
-    documents = open_documents(paths, proj.subjects,
-                               proj.vocab_lang, docs_limit)
+    documents = open_documents(paths, proj.subjects, proj.vocab_lang, docs_limit)
     proj.learn(documents, backend_params)
 
 
-@cli.command('suggest')
-@click.argument('project_id')
-@click.option('--limit', '-l', default=10, help='Maximum number of subjects')
-@click.option('--threshold', '-t', default=0.0, help='Minimum score threshold')
-@click.option('--language', '-L', help='Language of subject labels')
+@cli.command("suggest")
+@click.argument("project_id")
+@click.option("--limit", "-l", default=10, help="Maximum number of subjects")
+@click.option("--threshold", "-t", default=0.0, help="Minimum score threshold")
+@click.option("--language", "-L", help="Language of subject labels")
 @backend_param_option
 @common_options
 def run_suggest(project_id, limit, threshold, language, backend_param):
@@ -337,8 +356,7 @@ def run_suggest(project_id, limit, threshold, language, backend_param):
     text = sys.stdin.read()
     lang = language or project.vocab_lang
     if lang not in project.vocab.languages:
-        raise click.BadParameter(
-            f'language "{lang}" not supported by vocabulary')
+        raise click.BadParameter(f'language "{lang}" not supported by vocabulary')
     backend_params = parse_backend_params(backend_param, project)
     hit_filter = SuggestionFilter(project.subjects, limit, threshold)
     hits = hit_filter(project.suggest(text, backend_params))
@@ -347,29 +365,32 @@ def run_suggest(project_id, limit, threshold, language, backend_param):
         click.echo(
             "<{}>\t{}\t{}".format(
                 subj.uri,
-                '\t'.join(filter(None,
-                                 (subj.labels[lang],
-                                  subj.notation))),
-                hit.score))
+                "\t".join(filter(None, (subj.labels[lang], subj.notation))),
+                hit.score,
+            )
+        )
 
 
-@cli.command('index')
-@click.argument('project_id')
-@click.argument('directory', type=click.Path(exists=True, file_okay=False))
+@cli.command("index")
+@click.argument("project_id")
+@click.argument("directory", type=click.Path(exists=True, file_okay=False))
 @click.option(
-    '--suffix',
-    '-s',
-    default='.annif',
-    help='File name suffix for result files')
-@click.option('--force/--no-force', '-f/-F', default=False,
-              help='Force overwriting of existing result files')
-@click.option('--limit', '-l', default=10, help='Maximum number of subjects')
-@click.option('--threshold', '-t', default=0.0, help='Minimum score threshold')
-@click.option('--language', '-L', help='Language of subject labels')
+    "--suffix", "-s", default=".annif", help="File name suffix for result files"
+)
+@click.option(
+    "--force/--no-force",
+    "-f/-F",
+    default=False,
+    help="Force overwriting of existing result files",
+)
+@click.option("--limit", "-l", default=10, help="Maximum number of subjects")
+@click.option("--threshold", "-t", default=0.0, help="Minimum score threshold")
+@click.option("--language", "-L", help="Language of subject labels")
 @backend_param_option
 @common_options
-def run_index(project_id, directory, suffix, force,
-              limit, threshold, language, backend_param):
+def run_index(
+    project_id, directory, suffix, force, limit, threshold, language, backend_param
+):
     """
     Index a directory with documents, suggesting subjects for each document.
     Write the results in TSV files with the given suffix (``.annif`` by
@@ -378,81 +399,83 @@ def run_index(project_id, directory, suffix, force,
     project = get_project(project_id)
     lang = language or project.vocab_lang
     if lang not in project.vocab.languages:
-        raise click.BadParameter(
-            f'language "{lang}" not supported by vocabulary')
+        raise click.BadParameter(f'language "{lang}" not supported by vocabulary')
     backend_params = parse_backend_params(backend_param, project)
     hit_filter = SuggestionFilter(project.subjects, limit, threshold)
 
     for docfilename, dummy_subjectfn in annif.corpus.DocumentDirectory(
-            directory, project.subjects, project.vocab_lang,
-            require_subjects=False):
-        with open(docfilename, encoding='utf-8-sig') as docfile:
+        directory, project.subjects, project.vocab_lang, require_subjects=False
+    ):
+        with open(docfilename, encoding="utf-8-sig") as docfile:
             text = docfile.read()
-        subjectfilename = re.sub(r'\.txt$', suffix, docfilename)
+        subjectfilename = re.sub(r"\.txt$", suffix, docfilename)
         if os.path.exists(subjectfilename) and not force:
             click.echo(
-                "Not overwriting {} (use --force to override)".format(
-                    subjectfilename))
+                "Not overwriting {} (use --force to override)".format(subjectfilename)
+            )
             continue
-        with open(subjectfilename, 'w', encoding='utf-8') as subjfile:
+        with open(subjectfilename, "w", encoding="utf-8") as subjfile:
             results = project.suggest(text, backend_params)
             for hit in hit_filter(results).as_list():
                 subj = project.subjects[hit.subject_id]
                 line = "<{}>\t{}\t{}".format(
                     subj.uri,
-                    '\t'.join(filter(None, (subj.labels[lang],
-                                            subj.notation))),
-                    hit.score)
+                    "\t".join(filter(None, (subj.labels[lang], subj.notation))),
+                    hit.score,
+                )
                 click.echo(line, file=subjfile)
 
 
-@cli.command('eval')
-@click.argument('project_id')
-@click.argument('paths', type=click.Path(exists=True), nargs=-1)
-@click.option('--limit', '-l', default=10, help='Maximum number of subjects')
-@click.option('--threshold', '-t', default=0.0, help='Minimum score threshold')
-@click.option('--docs-limit', '-d', default=None,
-              type=click.IntRange(0, None),
-              help='Maximum number of documents to use')
-@click.option('--metric', '-m', default=[], multiple=True,
-              help='Metric to calculate (default: all)')
+@cli.command("eval")
+@click.argument("project_id")
+@click.argument("paths", type=click.Path(exists=True), nargs=-1)
+@click.option("--limit", "-l", default=10, help="Maximum number of subjects")
+@click.option("--threshold", "-t", default=0.0, help="Minimum score threshold")
+@click.option(
+    "--docs-limit",
+    "-d",
+    default=None,
+    type=click.IntRange(0, None),
+    help="Maximum number of documents to use",
+)
+@click.option(
+    "--metric",
+    "-m",
+    default=[],
+    multiple=True,
+    help="Metric to calculate (default: all)",
+)
 @click.option(
-    '--metrics-file',
-    '-M',
-    type=click.File(
-        'w',
-        encoding='utf-8',
-        errors='ignore',
-        lazy=True),
+    "--metrics-file",
+    "-M",
+    type=click.File("w", encoding="utf-8", errors="ignore", lazy=True),
     help="""Specify file in order to write evaluation metrics in JSON format.
-    File directory must exist, existing file will be overwritten.""")
+    File directory must exist, existing file will be overwritten.""",
+)
 @click.option(
-    '--results-file',
-    '-r',
-    type=click.File(
-        'w',
-        encoding='utf-8',
-        errors='ignore',
-        lazy=True),
+    "--results-file",
+    "-r",
+    type=click.File("w", encoding="utf-8", errors="ignore", lazy=True),
     help="""Specify file in order to write non-aggregated results per subject.
-    File directory must exist, existing file will be overwritten.""")
-@click.option('--jobs',
-              '-j',
-              default=1,
-              help='Number of parallel jobs (0 means all CPUs)')
+    File directory must exist, existing file will be overwritten.""",
+)
+@click.option(
+    "--jobs", "-j", default=1, help="Number of parallel jobs (0 means all CPUs)"
+)
 @backend_param_option
 @common_options
 def run_eval(
-        project_id,
-        paths,
-        limit,
-        threshold,
-        docs_limit,
-        metric,
-        metrics_file,
-        results_file,
-        jobs,
-        backend_param):
+    project_id,
+    paths,
+    limit,
+    threshold,
+    docs_limit,
+    metric,
+    metrics_file,
+    results_file,
+    jobs,
+    backend_param,
+):
     """
     Suggest subjects for documents and evaluate the results by comparing
     against a gold standard.
@@ -471,49 +494,58 @@ def run_eval(
     backend_params = parse_backend_params(backend_param, project)
 
     import annif.eval
+
     eval_batch = annif.eval.EvaluationBatch(project.subjects)
 
     if results_file:
         try:
-            print('', end='', file=results_file)
-            click.echo('Writing per subject evaluation results to {!s}'.format(
-                results_file.name))
+            print("", end="", file=results_file)
+            click.echo(
+                "Writing per subject evaluation results to {!s}".format(
+                    results_file.name
+                )
+            )
         except Exception as e:
             raise NotSupportedException(
-                "cannot open results-file for writing: " + str(e))
-    docs = open_documents(paths, project.subjects,
-                          project.vocab_lang, docs_limit)
+                "cannot open results-file for writing: " + str(e)
+            )
+    docs = open_documents(paths, project.subjects, project.vocab_lang, docs_limit)
 
     jobs, pool_class = annif.parallel.get_pool(jobs)
 
     project.initialize(parallel=True)
     psmap = annif.parallel.ProjectSuggestMap(
-        project.registry, [project_id], backend_params, limit, threshold)
+        project.registry, [project_id], backend_params, limit, threshold
+    )
 
     with pool_class(jobs) as pool:
-        for hits, subject_set in pool.imap_unordered(
-                psmap.suggest, docs.documents):
-            eval_batch.evaluate(hits[project_id],
-                                subject_set)
+        for hits, subject_set in pool.imap_unordered(psmap.suggest, docs.documents):
+            eval_batch.evaluate(hits[project_id], subject_set)
 
     template = "{0:<30}\t{1}"
-    metrics = eval_batch.results(metrics=metric,
-                                 results_file=results_file,
-                                 language=project.vocab_lang)
+    metrics = eval_batch.results(
+        metrics=metric, results_file=results_file, language=project.vocab_lang
+    )
     for metric, score in metrics.items():
         click.echo(template.format(metric + ":", score))
     if metrics_file:
         json.dump(
             {metric_code(mname): val for mname, val in metrics.items()},
-            metrics_file, indent=2)
+            metrics_file,
+            indent=2,
+        )
 
 
-@cli.command('optimize')
-@click.argument('project_id')
-@click.argument('paths', type=click.Path(exists=True), nargs=-1)
-@click.option('--docs-limit', '-d', default=None,
-              type=click.IntRange(0, None),
-              help='Maximum number of documents to use')
+@cli.command("optimize")
+@click.argument("project_id")
+@click.argument("paths", type=click.Path(exists=True), nargs=-1)
+@click.option(
+    "--docs-limit",
+    "-d",
+    default=None,
+    type=click.IntRange(0, None),
+    help="Maximum number of documents to use",
+)
 @backend_param_option
 @common_options
 def run_optimize(project_id, paths, docs_limit, backend_param):
@@ -533,19 +565,19 @@ def run_optimize(project_id, paths, docs_limit, backend_param):
     filter_batches = generate_filter_batches(project.subjects)
 
     ndocs = 0
-    docs = open_documents(paths, project.subjects,
-                          project.vocab_lang, docs_limit)
+    docs = open_documents(paths, project.subjects, project.vocab_lang, docs_limit)
     for doc in docs.documents:
         raw_hits = project.suggest(doc.text, backend_params)
         hits = raw_hits.filter(project.subjects, limit=BATCH_MAX_LIMIT)
-        assert isinstance(hits, ListSuggestionResult), \
-            "Optimize should only be done with ListSuggestionResult " + \
-            "as it would be very slow with VectorSuggestionResult."
+        assert isinstance(hits, ListSuggestionResult), (
+            "Optimize should only be done with ListSuggestionResult "
+            + "as it would be very slow with VectorSuggestionResult."
+        )
         for hit_filter, batch in filter_batches.values():
             batch.evaluate(hit_filter(hits), doc.subject_set)
         ndocs += 1
 
-    click.echo("\t".join(('Limit', 'Thresh.', 'Prec.', 'Rec.', 'F1')))
+    click.echo("\t".join(("Limit", "Thresh.", "Prec.", "Rec.", "F1")))
 
     best_scores = collections.defaultdict(float)
     best_params = {}
@@ -556,9 +588,7 @@ def run_optimize(project_id, paths, docs_limit, backend_param):
     filter_batches = list(filter_batches.items())
     while filter_batches:
         params, filter_batch = filter_batches.pop(0)
-        metrics = ['Precision (doc avg)',
-                   'Recall (doc avg)',
-                   'F1 score (doc avg)']
+        metrics = ["Precision (doc avg)", "Recall (doc avg)", "F1 score (doc avg)"]
         results = filter_batch[1].results(metrics=metrics)
         for metric, score in results.items():
             if score >= best_scores[metric]:
@@ -568,9 +598,11 @@ def run_optimize(project_id, paths, docs_limit, backend_param):
             template.format(
                 params[0],
                 params[1],
-                results['Precision (doc avg)'],
-                results['Recall (doc avg)'],
-                results['F1 score (doc avg)']))
+                results["Precision (doc avg)"],
+                results["Recall (doc avg)"],
+                results["F1 score (doc avg)"],
+            )
+        )
 
     click.echo()
     template2 = "Best {:>19}: {:.04f}\tLimit: {:d}\tThreshold: {:.02f}"
@@ -580,44 +612,45 @@ def run_optimize(project_id, paths, docs_limit, backend_param):
                 metric,
                 best_scores[metric],
                 best_params[metric][0],
-                best_params[metric][1]))
+                best_params[metric][1],
+            )
+        )
     click.echo("Documents evaluated:\t{}".format(ndocs))
 
 
-@cli.command('hyperopt')
-@click.argument('project_id')
-@click.argument('paths', type=click.Path(exists=True), nargs=-1)
-@click.option('--docs-limit', '-d', default=None,
-              type=click.IntRange(0, None),
-              help='Maximum number of documents to use')
-@click.option('--trials', '-T', default=10, help='Number of trials')
-@click.option('--jobs',
-              '-j',
-              default=1,
-              help='Number of parallel runs (0 means all CPUs)')
-@click.option('--metric', '-m', default='NDCG',
-              help='Metric to optimize (default: NDCG)')
+@cli.command("hyperopt")
+@click.argument("project_id")
+@click.argument("paths", type=click.Path(exists=True), nargs=-1)
+@click.option(
+    "--docs-limit",
+    "-d",
+    default=None,
+    type=click.IntRange(0, None),
+    help="Maximum number of documents to use",
+)
+@click.option("--trials", "-T", default=10, help="Number of trials")
+@click.option(
+    "--jobs", "-j", default=1, help="Number of parallel runs (0 means all CPUs)"
+)
+@click.option(
+    "--metric", "-m", default="NDCG", help="Metric to optimize (default: NDCG)"
+)
 @click.option(
-    '--results-file',
-    '-r',
-    type=click.File(
-        'w',
-        encoding='utf-8',
-        errors='ignore',
-        lazy=True),
+    "--results-file",
+    "-r",
+    type=click.File("w", encoding="utf-8", errors="ignore", lazy=True),
     help="""Specify file path to write trial results as CSV.
-    File directory must exist, existing file will be overwritten.""")
+    File directory must exist, existing file will be overwritten.""",
+)
 @common_options
-def run_hyperopt(project_id, paths, docs_limit, trials, jobs, metric,
-                 results_file):
+def run_hyperopt(project_id, paths, docs_limit, trials, jobs, metric, results_file):
     """
     Optimize the hyperparameters of a project using validation documents from
     ``PATHS``. Not supported by all backends. Output is a list of trial results
     and a report of the best performing parameters.
     """
     proj = get_project(project_id)
-    documents = open_documents(paths, proj.subjects,
-                               proj.vocab_lang, docs_limit)
+    documents = open_documents(paths, proj.subjects, proj.vocab_lang, docs_limit)
     click.echo(f"Looking for optimal hyperparameters using {trials} trials")
     rec = proj.hyperopt(documents, trials, jobs, metric, results_file)
     click.echo(f"Got best {metric} score {rec.score:.4f} with:")
@@ -627,5 +660,5 @@ def run_hyperopt(project_id, paths, docs_limit, trials, jobs, metric,
     click.echo("---")
 
 
-if __name__ == '__main__':
+if __name__ == "__main__":
     cli()
diff --git a/annif/config.py b/annif/config.py
index 951a40e26..838a8f906 100644
--- a/annif/config.py
+++ b/annif/config.py
@@ -19,13 +19,14 @@ class AnnifConfigCFG:
     def __init__(self, filename):
         self._config = configparser.ConfigParser()
         self._config.optionxform = annif.util.identity
-        with open(filename, encoding='utf-8-sig') as projf:
+        with open(filename, encoding="utf-8-sig") as projf:
             try:
-                logger.debug(
-                    f"Reading configuration file {filename} in CFG format")
+                logger.debug(f"Reading configuration file {filename} in CFG format")
                 self._config.read_file(projf)
-            except (configparser.DuplicateOptionError,
-                    configparser.DuplicateSectionError) as err:
+            except (
+                configparser.DuplicateOptionError,
+                configparser.DuplicateSectionError,
+            ) as err:
                 raise ConfigurationException(err)
 
     @property
@@ -42,12 +43,12 @@ class AnnifConfigTOML:
     def __init__(self, filename):
         with open(filename, "rb") as projf:
             try:
-                logger.debug(
-                    f"Reading configuration file {filename} in TOML format")
+                logger.debug(f"Reading configuration file {filename} in TOML format")
                 self._config = tomli.load(projf)
             except tomli.TOMLDecodeError as err:
                 raise ConfigurationException(
-                    f"Parsing TOML file '{filename}' failed: {err}")
+                    f"Parsing TOML file '{filename}' failed: {err}"
+                )
 
     @property
     def project_ids(self):
@@ -61,8 +62,8 @@ class AnnifConfigDirectory:
     """Class for reading configuration from directory"""
 
     def __init__(self, directory):
-        files = glob(os.path.join(directory, '*.cfg'))
-        files.extend(glob(os.path.join(directory, '*.toml')))
+        files = glob(os.path.join(directory, "*.cfg"))
+        files.extend(glob(os.path.join(directory, "*.toml")))
         logger.debug(f"Reading configuration files in directory {directory}")
 
         self._config = dict()
@@ -77,7 +78,8 @@ def _check_duplicate_project_ids(self, proj_id, file):
             # Error message resembles configparser's DuplicateSection message
             raise ConfigurationException(
                 f'While reading from "{file}": project ID "{proj_id}" already '
-                'exists in another configuration file in the directory.')
+                "exists in another configuration file in the directory."
+            )
 
     @property
     def project_ids(self):
@@ -92,25 +94,27 @@ def check_config(projects_config_path):
         return projects_config_path
     else:
         logger.warning(
-            'Project configuration file or directory ' +
-            f'"{projects_config_path}" is missing. Please provide one. ' +
-            'You can set the path to the project configuration ' +
-            'using the ANNIF_PROJECTS environment ' +
-            'variable or the command-line option "--projects".')
+            "Project configuration file or directory "
+            + f'"{projects_config_path}" is missing. Please provide one. '
+            + "You can set the path to the project configuration "
+            + "using the ANNIF_PROJECTS environment "
+            + 'variable or the command-line option "--projects".'
+        )
         return None
 
 
 def find_config():
-    for path in ('projects.cfg', 'projects.toml', 'projects.d'):
+    for path in ("projects.cfg", "projects.toml", "projects.d"):
         if os.path.exists(path):
             return path
 
     logger.warning(
-        'Could not find project configuration ' +
-        '"projects.cfg", "projects.toml" or "projects.d". ' +
-        'You can set the path to the project configuration ' +
-        'using the ANNIF_PROJECTS environment ' +
-        'variable or the command-line option "--projects".')
+        "Could not find project configuration "
+        + '"projects.cfg", "projects.toml" or "projects.d". '
+        + "You can set the path to the project configuration "
+        + "using the ANNIF_PROJECTS environment "
+        + 'variable or the command-line option "--projects".'
+    )
     return None
 
 
@@ -125,7 +129,7 @@ def parse_config(projects_config_path):
 
     if os.path.isdir(projects_config_path):
         return AnnifConfigDirectory(projects_config_path)
-    elif projects_config_path.endswith('.toml'):  # TOML format
+    elif projects_config_path.endswith(".toml"):  # TOML format
         return AnnifConfigTOML(projects_config_path)
     else:  # classic CFG/INI style format
         return AnnifConfigCFG(projects_config_path)
diff --git a/annif/corpus/__init__.py b/annif/corpus/__init__.py
index 88707fed4..6de290afd 100644
--- a/annif/corpus/__init__.py
+++ b/annif/corpus/__init__.py
@@ -1,15 +1,31 @@
 """Annif corpus operations"""
 
 
-from .document import DocumentDirectory, DocumentFile, DocumentList, \
-    TransformingDocumentCorpus, LimitingDocumentCorpus
+from .document import (
+    DocumentDirectory,
+    DocumentFile,
+    DocumentList,
+    TransformingDocumentCorpus,
+    LimitingDocumentCorpus,
+)
 from .subject import Subject, SubjectFileTSV, SubjectFileCSV
 from .subject import SubjectIndex, SubjectSet
 from .skos import SubjectFileSKOS
 from .types import Document
 from .combine import CombinedCorpus
 
-__all__ = ["DocumentDirectory", "DocumentFile", "DocumentList", "Subject",
-           "SubjectFileTSV", "SubjectFileCSV", "SubjectIndex", "SubjectSet",
-           "SubjectFileSKOS", "Document", "CombinedCorpus",
-           "TransformingDocumentCorpus", "LimitingDocumentCorpus"]
+__all__ = [
+    "DocumentDirectory",
+    "DocumentFile",
+    "DocumentList",
+    "Subject",
+    "SubjectFileTSV",
+    "SubjectFileCSV",
+    "SubjectIndex",
+    "SubjectSet",
+    "SubjectFileSKOS",
+    "Document",
+    "CombinedCorpus",
+    "TransformingDocumentCorpus",
+    "LimitingDocumentCorpus",
+]
diff --git a/annif/corpus/combine.py b/annif/corpus/combine.py
index 8dbdee611..73c217b0e 100644
--- a/annif/corpus/combine.py
+++ b/annif/corpus/combine.py
@@ -14,4 +14,5 @@ def __init__(self, corpora):
     @property
     def documents(self):
         return itertools.chain.from_iterable(
-            [corpus.documents for corpus in self._corpora])
+            [corpus.documents for corpus in self._corpora]
+        )
diff --git a/annif/corpus/document.py b/annif/corpus/document.py
index b966c56ec..c28ae0b07 100644
--- a/annif/corpus/document.py
+++ b/annif/corpus/document.py
@@ -26,12 +26,12 @@ def __iter__(self):
         subjectfile) containing file paths. If there is no key file and
         require_subjects is False, the subjectfile will be returned as None."""
 
-        for filename in sorted(glob.glob(os.path.join(self.path, '*.txt'))):
-            tsvfilename = re.sub(r'\.txt$', '.tsv', filename)
+        for filename in sorted(glob.glob(os.path.join(self.path, "*.txt"))):
+            tsvfilename = re.sub(r"\.txt$", ".tsv", filename)
             if os.path.exists(tsvfilename):
                 yield (filename, tsvfilename)
                 continue
-            keyfilename = re.sub(r'\.txt$', '.key', filename)
+            keyfilename = re.sub(r"\.txt$", ".key", filename)
             if os.path.exists(keyfilename):
                 yield (filename, keyfilename)
                 continue
@@ -41,13 +41,12 @@ def __iter__(self):
     @property
     def documents(self):
         for docfilename, keyfilename in self:
-            with open(docfilename, errors='replace',
-                      encoding='utf-8-sig') as docfile:
+            with open(docfilename, errors="replace", encoding="utf-8-sig") as docfile:
                 text = docfile.read()
-            with open(keyfilename, encoding='utf-8-sig') as keyfile:
-                subjects = SubjectSet.from_string(keyfile.read(),
-                                                  self.subject_index,
-                                                  self.language)
+            with open(keyfilename, encoding="utf-8-sig") as keyfile:
+                subjects = SubjectSet.from_string(
+                    keyfile.read(), self.subject_index, self.language
+                )
             yield Document(text=text, subject_set=subjects)
 
 
@@ -60,25 +59,24 @@ def __init__(self, path, subject_index):
 
     @property
     def documents(self):
-        if self.path.endswith('.gz'):
+        if self.path.endswith(".gz"):
             opener = gzip.open
         else:
             opener = open
-        with opener(self.path, mode='rt', encoding='utf-8-sig') as tsvfile:
+        with opener(self.path, mode="rt", encoding="utf-8-sig") as tsvfile:
             for line in tsvfile:
                 yield from self._parse_tsv_line(line)
 
     def _parse_tsv_line(self, line):
-        if '\t' in line:
-            text, uris = line.split('\t', maxsplit=1)
-            subject_ids = {self.subject_index.by_uri(
-                               annif.util.cleanup_uri(uri))
-                           for uri in uris.split()}
-            yield Document(text=text,
-                           subject_set=SubjectSet(subject_ids))
+        if "\t" in line:
+            text, uris = line.split("\t", maxsplit=1)
+            subject_ids = {
+                self.subject_index.by_uri(annif.util.cleanup_uri(uri))
+                for uri in uris.split()
+            }
+            yield Document(text=text, subject_set=SubjectSet(subject_ids))
         else:
-            logger.warning('Skipping invalid line (missing tab): "%s"',
-                           line.rstrip())
+            logger.warning('Skipping invalid line (missing tab): "%s"', line.rstrip())
 
 
 class DocumentList(DocumentCorpus):
@@ -104,8 +102,9 @@ def __init__(self, corpus, transform_fn):
     @property
     def documents(self):
         for doc in self._orig_corpus.documents:
-            yield Document(text=self._transform_fn(doc.text),
-                           subject_set=doc.subject_set)
+            yield Document(
+                text=self._transform_fn(doc.text), subject_set=doc.subject_set
+            )
 
 
 class LimitingDocumentCorpus(DocumentCorpus):
diff --git a/annif/corpus/skos.py b/annif/corpus/skos.py
index 95702a326..4b18f58f3 100644
--- a/annif/corpus/skos.py
+++ b/annif/corpus/skos.py
@@ -16,21 +16,29 @@ def serialize_subjects_to_skos(subjects, path):
     into a SKOS/Turtle file with the given path name."""
 
     graph = rdflib.Graph()
-    graph.namespace_manager.bind('skos', SKOS)
+    graph.namespace_manager.bind("skos", SKOS)
     for subject in subjects:
         graph.add((rdflib.URIRef(subject.uri), RDF.type, SKOS.Concept))
         for lang, label in subject.labels.items():
-            graph.add((rdflib.URIRef(subject.uri),
-                       SKOS.prefLabel,
-                       rdflib.Literal(label, lang)))
-        graph.add((rdflib.URIRef(subject.uri),
-                   SKOS.notation,
-                   rdflib.Literal(subject.notation)))
-    graph.serialize(destination=path, format='turtle')
+            graph.add(
+                (
+                    rdflib.URIRef(subject.uri),
+                    SKOS.prefLabel,
+                    rdflib.Literal(label, lang),
+                )
+            )
+        graph.add(
+            (
+                rdflib.URIRef(subject.uri),
+                SKOS.notation,
+                rdflib.Literal(subject.notation),
+            )
+        )
+    graph.serialize(destination=path, format="turtle")
     # also dump the graph in joblib format which is faster to load
-    annif.util.atomic_save(graph,
-                           *os.path.split(path.replace('.ttl', '.dump.gz')),
-                           method=joblib.dump)
+    annif.util.atomic_save(
+        graph, *os.path.split(path.replace(".ttl", ".dump.gz")), method=joblib.dump
+    )
 
 
 class SubjectFileSKOS(SubjectCorpus):
@@ -42,31 +50,34 @@ class SubjectFileSKOS(SubjectCorpus):
 
     def __init__(self, path):
         self.path = path
-        if path.endswith('.dump.gz'):
+        if path.endswith(".dump.gz"):
             self.graph = joblib.load(path)
         else:
             self.graph = rdflib.Graph()
-            self.graph.parse(self.path,
-                             format=rdflib.util.guess_format(self.path))
+            self.graph.parse(self.path, format=rdflib.util.guess_format(self.path))
 
     @property
     def languages(self):
         if self._languages is None:
-            self._languages = {label.language
-                               for concept in self.concepts
-                               for label_type in self.PREF_LABEL_PROPERTIES
-                               for label in self.graph.objects(concept,
-                                                               label_type)
-                               if label.language is not None}
+            self._languages = {
+                label.language
+                for concept in self.concepts
+                for label_type in self.PREF_LABEL_PROPERTIES
+                for label in self.graph.objects(concept, label_type)
+                if label.language is not None
+            }
         return self._languages
 
     def _concept_labels(self, concept):
-        by_lang = self.get_concept_labels(concept,
-                                          self.PREF_LABEL_PROPERTIES)
-        return {lang: by_lang[lang][0] if by_lang[lang]  # correct lang
-                else by_lang[None][0] if by_lang[None]  # no language
-                else self.graph.namespace_manager.qname(concept)
-                for lang in self.languages}
+        by_lang = self.get_concept_labels(concept, self.PREF_LABEL_PROPERTIES)
+        return {
+            lang: by_lang[lang][0]
+            if by_lang[lang]  # correct lang
+            else by_lang[None][0]
+            if by_lang[None]  # no language
+            else self.graph.namespace_manager.qname(concept)
+            for lang in self.languages
+        }
 
     @property
     def subjects(self):
@@ -110,16 +121,16 @@ def save_skos(self, path):
         """Save the contents of the subject vocabulary into a SKOS/Turtle
         file with the given path name."""
 
-        if self.path.endswith('.ttl'):
+        if self.path.endswith(".ttl"):
             # input is already in Turtle syntax, no need to reserialize
-            if not os.path.exists(path) or \
-               not os.path.samefile(self.path, path):
+            if not os.path.exists(path) or not os.path.samefile(self.path, path):
                 shutil.copyfile(self.path, path)
         else:
             # need to serialize into Turtle
-            self.graph.serialize(destination=path, format='turtle')
+            self.graph.serialize(destination=path, format="turtle")
         # also dump the graph in joblib format which is faster to load
-        annif.util.atomic_save(self.graph,
-                               *os.path.split(
-                                   path.replace('.ttl', '.dump.gz')),
-                               method=joblib.dump)
+        annif.util.atomic_save(
+            self.graph,
+            *os.path.split(path.replace(".ttl", ".dump.gz")),
+            method=joblib.dump
+        )
diff --git a/annif/corpus/subject.py b/annif/corpus/subject.py
index e380d190a..677caee8e 100644
--- a/annif/corpus/subject.py
+++ b/annif/corpus/subject.py
@@ -20,14 +20,12 @@ def __init__(self, path, language):
         self.language = language
 
     def _parse_line(self, line):
-        vals = line.strip().split('\t', 2)
+        vals = line.strip().split("\t", 2)
         clean_uri = annif.util.cleanup_uri(vals[0])
         label = vals[1] if len(vals) >= 2 else None
         labels = {self.language: label} if label else None
         notation = vals[2] if len(vals) >= 3 else None
-        yield Subject(uri=clean_uri,
-                      labels=labels,
-                      notation=notation)
+        yield Subject(uri=clean_uri, labels=labels, notation=notation)
 
     @property
     def languages(self):
@@ -35,7 +33,7 @@ def languages(self):
 
     @property
     def subjects(self):
-        with open(self.path, encoding='utf-8-sig') as subjfile:
+        with open(self.path, encoding="utf-8-sig") as subjfile:
             for line in subjfile:
                 yield from self._parse_line(line)
 
@@ -54,9 +52,9 @@ def __init__(self, path):
 
     def _parse_row(self, row):
         labels = {
-            fname.replace('label_', ''): value or None
+            fname.replace("label_", ""): value or None
             for fname, value in row.items()
-            if fname.startswith('label_')
+            if fname.startswith("label_")
         }
 
         # if there are no labels in any language, set labels to None
@@ -64,24 +62,28 @@ def _parse_row(self, row):
         if set(labels.values()) == {None}:
             labels = None
 
-        yield Subject(uri=annif.util.cleanup_uri(row['uri']),
-                      labels=labels,
-                      notation=row.get('notation', None) or None)
+        yield Subject(
+            uri=annif.util.cleanup_uri(row["uri"]),
+            labels=labels,
+            notation=row.get("notation", None) or None,
+        )
 
     @property
     def languages(self):
         # infer the supported languages from the CSV column names
-        with open(self.path, encoding='utf-8-sig') as csvfile:
+        with open(self.path, encoding="utf-8-sig") as csvfile:
             reader = csv.reader(csvfile)
             fieldnames = next(reader, None)
 
-        return [fname.replace('label_', '')
-                for fname in fieldnames
-                if fname.startswith('label_')]
+        return [
+            fname.replace("label_", "")
+            for fname in fieldnames
+            if fname.startswith("label_")
+        ]
 
     @property
     def subjects(self):
-        with open(self.path, encoding='utf-8-sig') as csvfile:
+        with open(self.path, encoding="utf-8-sig") as csvfile:
             reader = csv.DictReader(csvfile)
             for row in reader:
                 yield from self._parse_row(row)
@@ -95,7 +97,7 @@ def save_skos(self, path):
     def is_csv_file(path):
         """return True if the path looks like a CSV file"""
 
-        return os.path.splitext(path)[1].lower() == '.csv'
+        return os.path.splitext(path)[1].lower() == ".csv"
 
 
 class SubjectIndex:
@@ -146,7 +148,7 @@ def by_uri(self, uri, warnings=True):
             return self._uri_idx[uri]
         except KeyError:
             if warnings:
-                logger.warning('Unknown subject URI <%s>', uri)
+                logger.warning("Unknown subject URI <%s>", uri)
             return None
 
     def by_label(self, label, language):
@@ -161,34 +163,36 @@ def by_label(self, label, language):
     def deprecated_ids(self):
         """return indices of deprecated subjects"""
 
-        return [subject_id for subject_id, subject in enumerate(self._subjects)
-                if subject.labels is None]
+        return [
+            subject_id
+            for subject_id, subject in enumerate(self._subjects)
+            if subject.labels is None
+        ]
 
     @property
     def active(self):
         """return a list of (subject_id, subject) tuples of all subjects that
         are not deprecated"""
 
-        return [(subj_id, subject)
-                for subj_id, subject
-                in enumerate(self._subjects)
-                if subject.labels is not None]
+        return [
+            (subj_id, subject)
+            for subj_id, subject in enumerate(self._subjects)
+            if subject.labels is not None
+        ]
 
     def save(self, path):
         """Save this subject index into a file with the given path name."""
 
-        fieldnames = ['uri', 'notation'] + \
-            [f'label_{lang}' for lang in self._languages]
+        fieldnames = ["uri", "notation"] + [f"label_{lang}" for lang in self._languages]
 
-        with open(path, 'w', encoding='utf-8', newline='') as csvfile:
+        with open(path, "w", encoding="utf-8", newline="") as csvfile:
             writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
             writer.writeheader()
             for subject in self:
-                row = {'uri': subject.uri,
-                       'notation': subject.notation or ''}
+                row = {"uri": subject.uri, "notation": subject.notation or ""}
                 if subject.labels:
                     for lang, label in subject.labels.items():
-                        row[f'label_{lang}'] = label
+                        row[f"label_{lang}"] = label
                 writer.writerow(row)
 
     @classmethod
@@ -210,9 +214,9 @@ def __init__(self, subject_ids=None):
 
         if subject_ids:
             # use set comprehension to eliminate possible duplicates
-            self._subject_ids = list({subject_id
-                                      for subject_id in subject_ids
-                                      if subject_id is not None})
+            self._subject_ids = list(
+                {subject_id for subject_id in subject_ids if subject_id is not None}
+            )
         else:
             self._subject_ids = []
 
@@ -248,9 +252,9 @@ def _parse_line(line):
         vals = line.split("\t")
         for val in vals:
             val = val.strip()
-            if val == '':
+            if val == "":
                 continue
-            if val.startswith('<') and val.endswith('>'):  # URI
+            if val.startswith("<") and val.endswith(">"):  # URI
                 uri = val[1:-1]
                 continue
             label = val
@@ -259,8 +263,8 @@ def _parse_line(line):
 
     def as_vector(self, size=None, destination=None):
         """Return the hits as a one-dimensional NumPy array in sklearn
-           multilabel indicator format. Use destination array if given (not
-           None), otherwise create and return a new one of the given size."""
+        multilabel indicator format. Use destination array if given (not
+        None), otherwise create and return a new one of the given size."""
 
         if destination is None:
             destination = np.zeros(size, dtype=bool)
diff --git a/annif/corpus/types.py b/annif/corpus/types.py
index 018e4308e..9b8577741 100644
--- a/annif/corpus/types.py
+++ b/annif/corpus/types.py
@@ -4,7 +4,7 @@
 import collections
 
 
-Document = collections.namedtuple('Document', 'text subject_set')
+Document = collections.namedtuple("Document", "text subject_set")
 
 
 class DocumentCorpus(metaclass=abc.ABCMeta):
@@ -25,7 +25,7 @@ def is_empty(self):
             return True
 
 
-Subject = collections.namedtuple('Subject', 'uri labels notation')
+Subject = collections.namedtuple("Subject", "uri labels notation")
 
 
 class SubjectCorpus(metaclass=abc.ABCMeta):
diff --git a/annif/default_config.py b/annif/default_config.py
index 3e5c5ee2e..dfbf8e1be 100644
--- a/annif/default_config.py
+++ b/annif/default_config.py
@@ -11,8 +11,8 @@
 class Config(object):
     DEBUG = False
     TESTING = False
-    PROJECTS_CONFIG_PATH = os.environ.get('ANNIF_PROJECTS', default='')
-    DATADIR = os.environ.get('ANNIF_DATADIR', default='data')
+    PROJECTS_CONFIG_PATH = os.environ.get("ANNIF_PROJECTS", default="")
+    DATADIR = os.environ.get("ANNIF_DATADIR", default="data")
     INITIALIZE_PROJECTS = False
 
 
@@ -26,8 +26,8 @@ class DevelopmentConfig(Config):
 
 class TestingConfig(Config):
     TESTING = True
-    PROJECTS_CONFIG_PATH = 'tests/projects.cfg'
-    DATADIR = 'tests/data'
+    PROJECTS_CONFIG_PATH = "tests/projects.cfg"
+    DATADIR = "tests/data"
 
 
 class TestingInitializeConfig(TestingConfig):
@@ -35,16 +35,16 @@ class TestingInitializeConfig(TestingConfig):
 
 
 class TestingNoProjectsConfig(TestingConfig):
-    PROJECTS_CONFIG_PATH = 'tests/notfound.cfg'
+    PROJECTS_CONFIG_PATH = "tests/notfound.cfg"
 
 
 class TestingInvalidProjectsConfig(TestingConfig):
-    PROJECTS_CONFIG_PATH = 'tests/projects_invalid.cfg'
+    PROJECTS_CONFIG_PATH = "tests/projects_invalid.cfg"
 
 
 class TestingTOMLConfig(TestingConfig):
-    PROJECTS_CONFIG_PATH = 'tests/projects.toml'
+    PROJECTS_CONFIG_PATH = "tests/projects.toml"
 
 
 class TestingDirectoryConfig(TestingConfig):
-    PROJECTS_CONFIG_PATH = 'tests/projects.d'
+    PROJECTS_CONFIG_PATH = "tests/projects.d"
diff --git a/annif/eval.py b/annif/eval.py
index 149982781..fc8ac6826 100644
--- a/annif/eval.py
+++ b/annif/eval.py
@@ -103,105 +103,115 @@ def _evaluate_samples(self, y_true, y_pred, metrics=[]):
         # define the available metrics as lazy lambda functions
         # so we can execute only the ones actually requested
         all_metrics = {
-            'Precision (doc avg)': lambda: precision_score(
-                y_true_sparse, y_pred_binary, average='samples'),
-            'Recall (doc avg)': lambda: recall_score(
-                y_true_sparse, y_pred_binary, average='samples'),
-            'F1 score (doc avg)': lambda: f1_score(
-                y_true_sparse, y_pred_binary, average='samples'),
-            'Precision (subj avg)': lambda: precision_score(
-                y_true_sparse, y_pred_binary, average='macro'),
-            'Recall (subj avg)': lambda: recall_score(
-                y_true_sparse, y_pred_binary, average='macro'),
-            'F1 score (subj avg)': lambda: f1_score(
-                y_true_sparse, y_pred_binary, average='macro'),
-            'Precision (weighted subj avg)': lambda: precision_score(
-                y_true_sparse, y_pred_binary, average='weighted'),
-            'Recall (weighted subj avg)': lambda: recall_score(
-                y_true_sparse, y_pred_binary, average='weighted'),
-            'F1 score (weighted subj avg)': lambda: f1_score(
-                y_true_sparse, y_pred_binary, average='weighted'),
-            'Precision (microavg)': lambda: precision_score(
-                y_true_sparse, y_pred_binary, average='micro'),
-            'Recall (microavg)': lambda: recall_score(
-                y_true_sparse, y_pred_binary, average='micro'),
-            'F1 score (microavg)': lambda: f1_score(
-                y_true_sparse, y_pred_binary, average='micro'),
-            'F1@5': lambda: f1_score(
-                y_true_sparse,
-                filter_pred_top_k(y_pred, 5) > 0.0,
-                average='samples'),
-            'NDCG': lambda: ndcg_score(y_true, y_pred),
-            'NDCG@5': lambda: ndcg_score(y_true, y_pred, limit=5),
-            'NDCG@10': lambda: ndcg_score(y_true, y_pred, limit=10),
-            'Precision@1': lambda: precision_at_k_score(
-                y_true, y_pred, limit=1),
-            'Precision@3': lambda: precision_at_k_score(
-                y_true, y_pred, limit=3),
-            'Precision@5': lambda: precision_at_k_score(
-                y_true, y_pred, limit=5),
-            'LRAP': lambda: label_ranking_average_precision_score(
-                y_true, y_pred),
-            'True positives': lambda: true_positives(
-                y_true, y_pred_binary),
-            'False positives': lambda: false_positives(
-                y_true, y_pred_binary),
-            'False negatives': lambda: false_negatives(
-                y_true, y_pred_binary),
+            "Precision (doc avg)": lambda: precision_score(
+                y_true_sparse, y_pred_binary, average="samples"
+            ),
+            "Recall (doc avg)": lambda: recall_score(
+                y_true_sparse, y_pred_binary, average="samples"
+            ),
+            "F1 score (doc avg)": lambda: f1_score(
+                y_true_sparse, y_pred_binary, average="samples"
+            ),
+            "Precision (subj avg)": lambda: precision_score(
+                y_true_sparse, y_pred_binary, average="macro"
+            ),
+            "Recall (subj avg)": lambda: recall_score(
+                y_true_sparse, y_pred_binary, average="macro"
+            ),
+            "F1 score (subj avg)": lambda: f1_score(
+                y_true_sparse, y_pred_binary, average="macro"
+            ),
+            "Precision (weighted subj avg)": lambda: precision_score(
+                y_true_sparse, y_pred_binary, average="weighted"
+            ),
+            "Recall (weighted subj avg)": lambda: recall_score(
+                y_true_sparse, y_pred_binary, average="weighted"
+            ),
+            "F1 score (weighted subj avg)": lambda: f1_score(
+                y_true_sparse, y_pred_binary, average="weighted"
+            ),
+            "Precision (microavg)": lambda: precision_score(
+                y_true_sparse, y_pred_binary, average="micro"
+            ),
+            "Recall (microavg)": lambda: recall_score(
+                y_true_sparse, y_pred_binary, average="micro"
+            ),
+            "F1 score (microavg)": lambda: f1_score(
+                y_true_sparse, y_pred_binary, average="micro"
+            ),
+            "F1@5": lambda: f1_score(
+                y_true_sparse, filter_pred_top_k(y_pred, 5) > 0.0, average="samples"
+            ),
+            "NDCG": lambda: ndcg_score(y_true, y_pred),
+            "NDCG@5": lambda: ndcg_score(y_true, y_pred, limit=5),
+            "NDCG@10": lambda: ndcg_score(y_true, y_pred, limit=10),
+            "Precision@1": lambda: precision_at_k_score(y_true, y_pred, limit=1),
+            "Precision@3": lambda: precision_at_k_score(y_true, y_pred, limit=3),
+            "Precision@5": lambda: precision_at_k_score(y_true, y_pred, limit=5),
+            "LRAP": lambda: label_ranking_average_precision_score(y_true, y_pred),
+            "True positives": lambda: true_positives(y_true, y_pred_binary),
+            "False positives": lambda: false_positives(y_true, y_pred_binary),
+            "False negatives": lambda: false_negatives(y_true, y_pred_binary),
         }
 
         if not metrics:
             metrics = all_metrics.keys()
 
         with warnings.catch_warnings():
-            warnings.simplefilter('ignore')
+            warnings.simplefilter("ignore")
 
             return {metric: all_metrics[metric]() for metric in metrics}
 
     def _result_per_subject_header(self, results_file):
-        print('\t'.join(['URI',
-                         'Label',
-                         'Support',
-                         'True_positives',
-                         'False_positives',
-                         'False_negatives',
-                         'Precision',
-                         'Recall',
-                         'F1_score']),
-              file=results_file)
+        print(
+            "\t".join(
+                [
+                    "URI",
+                    "Label",
+                    "Support",
+                    "True_positives",
+                    "False_positives",
+                    "False_negatives",
+                    "Precision",
+                    "Recall",
+                    "F1_score",
+                ]
+            ),
+            file=results_file,
+        )
 
     def _result_per_subject_body(self, zipped_results, results_file):
         for row in zipped_results:
-            print('\t'.join((str(e) for e in row)), file=results_file)
+            print("\t".join((str(e) for e in row)), file=results_file)
 
-    def output_result_per_subject(self, y_true, y_pred,
-                                  results_file, language):
+    def output_result_per_subject(self, y_true, y_pred, results_file, language):
         """Write results per subject (non-aggregated)
         to outputfile results_file, using labels in the given language"""
 
         y_pred = y_pred.T > 0.0
         y_true = y_true.T > 0.0
 
-        true_pos = (y_true & y_pred)
-        false_pos = (~y_true & y_pred)
-        false_neg = (y_true & ~y_pred)
+        true_pos = y_true & y_pred
+        false_pos = ~y_true & y_pred
+        false_neg = y_true & ~y_pred
 
         r = len(y_true)
 
-        zipped = zip([subj.uri
-                      for subj in self._subject_index],       # URI
-                     [subj.labels[language]
-                      for subj in self._subject_index],	      # Label
-                     np.sum((true_pos + false_neg), axis=1),  # Support
-                     np.sum(true_pos, axis=1),                # True_positives
-                     np.sum(false_pos, axis=1),               # False_positives
-                     np.sum(false_neg, axis=1),               # False_negatives
-                     [precision_score(y_true[i], y_pred[i], zero_division=0)
-                      for i in range(r)],                     # Precision
-                     [recall_score(y_true[i], y_pred[i], zero_division=0)
-                      for i in range(r)],                     # Recall
-                     [f1_score(y_true[i], y_pred[i], zero_division=0)
-                      for i in range(r)])                     # F1
+        zipped = zip(
+            [subj.uri for subj in self._subject_index],  # URI
+            [subj.labels[language] for subj in self._subject_index],  # Label
+            np.sum((true_pos + false_neg), axis=1),  # Support
+            np.sum(true_pos, axis=1),  # True_positives
+            np.sum(false_pos, axis=1),  # False_positives
+            np.sum(false_neg, axis=1),  # False_negatives
+            [
+                precision_score(y_true[i], y_pred[i], zero_division=0) for i in range(r)
+            ],  # Precision
+            [
+                recall_score(y_true[i], y_pred[i], zero_division=0) for i in range(r)
+            ],  # Recall
+            [f1_score(y_true[i], y_pred[i], zero_division=0) for i in range(r)],
+        )  # F1
         self._result_per_subject_header(results_file)
         self._result_per_subject_body(zipped, results_file)
 
@@ -223,9 +233,8 @@ def results(self, metrics=[], results_file=None, language=None):
             hits.as_vector(len(self._subject_index), destination=y_pred[idx])
 
         results = self._evaluate_samples(y_true, y_pred, metrics)
-        results['Documents evaluated'] = int(y_true.shape[0])
+        results["Documents evaluated"] = int(y_true.shape[0])
 
         if results_file:
-            self.output_result_per_subject(y_true, y_pred,
-                                           results_file, language)
+            self.output_result_per_subject(y_true, y_pred, results_file, language)
         return results
diff --git a/annif/exception.py b/annif/exception.py
index 1ce754ca7..efc2d4a3e 100644
--- a/annif/exception.py
+++ b/annif/exception.py
@@ -22,13 +22,13 @@ def __init__(self, message, project_id=None, backend_id=None):
 
     def format_message(self):
         if self.project_id is not None:
-            return "{} project '{}': {}".format(self.prefix,
-                                                self.project_id,
-                                                self.message)
+            return "{} project '{}': {}".format(
+                self.prefix, self.project_id, self.message
+            )
         if self.backend_id is not None:
-            return "{} backend '{}': {}".format(self.prefix,
-                                                self.backend_id,
-                                                self.message)
+            return "{} backend '{}': {}".format(
+                self.prefix, self.backend_id, self.message
+            )
         return "{}: {}".format(self.prefix, self.message)
 
 
diff --git a/annif/lexical/mllm.py b/annif/lexical/mllm.py
index fbb443f3f..e075e02ae 100644
--- a/annif/lexical/mllm.py
+++ b/annif/lexical/mllm.py
@@ -18,27 +18,27 @@
 from annif.lexical.util import make_relation_matrix, make_collection_matrix
 
 
-Term = collections.namedtuple('Term', 'subject_id label is_pref')
+Term = collections.namedtuple("Term", "subject_id label is_pref")
 
-Match = collections.namedtuple(
-    'Match', 'subject_id is_pref n_tokens pos ambiguity')
+Match = collections.namedtuple("Match", "subject_id is_pref n_tokens pos ambiguity")
 
 Candidate = collections.namedtuple(
-    'Candidate',
-    'doc_length subject_id freq is_pref n_tokens ambiguity ' +
-    'first_occ last_occ spread')
+    "Candidate",
+    "doc_length subject_id freq is_pref n_tokens ambiguity "
+    + "first_occ last_occ spread",
+)
 
 ModelData = collections.namedtuple(
-    'ModelData',
-    'broader narrower related collection ' +
-    'doc_freq subj_freq idf')
+    "ModelData", "broader narrower related collection " + "doc_freq subj_freq idf"
+)
 
 Feature = IntEnum(
-    'Feature',
-    'freq doc_freq subj_freq tfidf is_pref n_tokens ambiguity ' +
-    'first_occ last_occ spread doc_length ' +
-    'broader narrower related collection',
-    start=0)
+    "Feature",
+    "freq doc_freq subj_freq tfidf is_pref n_tokens ambiguity "
+    + "first_occ last_occ spread doc_length "
+    + "broader narrower related collection",
+    start=0,
+)
 
 
 def conflate_matches(matches, doc_length):
@@ -55,9 +55,10 @@ def conflate_matches(matches, doc_length):
             ambiguity=mean((m.ambiguity for m in matches)),
             first_occ=matches[0].pos / doc_length,
             last_occ=matches[-1].pos / doc_length,
-            spread=(matches[-1].pos - matches[0].pos) / doc_length
+            spread=(matches[-1].pos - matches[0].pos) / doc_length,
         )
-        for subject_id, matches in subj_matches.items()]
+        for subject_id, matches in subj_matches.items()
+    ]
 
 
 def generate_candidates(text, analyzer, vectorizer, index):
@@ -68,11 +69,15 @@ def generate_candidates(text, analyzer, vectorizer, index):
     for sent_idx, token_matrix in enumerate(sent_tokens):
         tset = TokenSet(token_matrix.nonzero()[1])
         for ts, ambiguity in index.search(tset):
-            matches.append(Match(subject_id=ts.subject_id,
-                                 is_pref=ts.is_pref,
-                                 n_tokens=len(ts),
-                                 pos=sent_idx,
-                                 ambiguity=ambiguity))
+            matches.append(
+                Match(
+                    subject_id=ts.subject_id,
+                    is_pref=ts.is_pref,
+                    n_tokens=len(ts),
+                    pos=sent_idx,
+                    ambiguity=ambiguity,
+                )
+            )
 
     return conflate_matches(matches, len(sentences))
 
@@ -87,8 +92,7 @@ def candidates_to_features(candidates, mdata):
     broader = mdata.broader.multiply(c_vec).sum(axis=1)
     narrower = mdata.narrower.multiply(c_vec).sum(axis=1)
     related = mdata.related.multiply(c_vec).sum(axis=1)
-    collection = mdata.collection.multiply(c_vec).T.dot(
-        mdata.collection).sum(axis=0)
+    collection = mdata.collection.multiply(c_vec).T.dot(mdata.collection).sum(axis=0)
     for idx, c in enumerate(candidates):
         subj = c.subject_id
         matrix[idx, Feature.freq] = c.freq
@@ -110,7 +114,6 @@ def candidates_to_features(candidates, mdata):
 
 
 class MLLMCandidateGenerator(annif.parallel.BaseWorker):
-
     @classmethod
     def generate_candidates(cls, doc_subject_set, text):
         candidates = generate_candidates(text, **cls.args)  # pragma: no cover
@@ -118,29 +121,28 @@ def generate_candidates(cls, doc_subject_set, text):
 
 
 class MLLMFeatureConverter(annif.parallel.BaseWorker):
-
     @classmethod
     def candidates_to_features(cls, candidates):
-        return candidates_to_features(candidates,
-                                      **cls.args)  # pragma: no cover
+        return candidates_to_features(candidates, **cls.args)  # pragma: no cover
 
 
 class MLLMModel:
     """Maui-like Lexical Matching model"""
 
     def generate_candidates(self, text, analyzer):
-        return generate_candidates(text, analyzer,
-                                   self._vectorizer, self._index)
+        return generate_candidates(text, analyzer, self._vectorizer, self._index)
 
     @property
     def _model_data(self):
-        return ModelData(broader=self._broader_matrix,
-                         narrower=self._narrower_matrix,
-                         related=self._related_matrix,
-                         collection=self._collection_matrix,
-                         doc_freq=self._doc_freq,
-                         subj_freq=self._subj_freq,
-                         idf=self._idf)
+        return ModelData(
+            broader=self._broader_matrix,
+            narrower=self._narrower_matrix,
+            related=self._related_matrix,
+            collection=self._collection_matrix,
+            doc_freq=self._doc_freq,
+            subj_freq=self._subj_freq,
+            idf=self._idf,
+        )
 
     def _candidates_to_features(self, candidates):
         return candidates_to_features(candidates, self._model_data)
@@ -149,7 +151,7 @@ def _candidates_to_features(self, candidates):
     def _get_label_props(params):
         pref_label_props = [SKOS.prefLabel]
 
-        if annif.util.boolean(params['use_hidden_labels']):
+        if annif.util.boolean(params["use_hidden_labels"]):
             nonpref_label_props = [SKOS.altLabel, SKOS.hiddenLabel]
         else:
             nonpref_label_props = [SKOS.altLabel]
@@ -164,29 +166,22 @@ def _prepare_terms(self, graph, vocab, params):
         for subj_id, subject in vocab.subjects.active:
             subject_ids.append(subj_id)
 
-            for label in get_subject_labels(graph, subject.uri,
-                                            pref_label_props,
-                                            params['language']):
-                terms.append(Term(subject_id=subj_id,
-                                  label=label,
-                                  is_pref=True))
+            for label in get_subject_labels(
+                graph, subject.uri, pref_label_props, params["language"]
+            ):
+                terms.append(Term(subject_id=subj_id, label=label, is_pref=True))
 
-            for label in get_subject_labels(graph, subject.uri,
-                                            nonpref_label_props,
-                                            params['language']):
-                terms.append(Term(subject_id=subj_id,
-                                  label=label,
-                                  is_pref=False))
+            for label in get_subject_labels(
+                graph, subject.uri, nonpref_label_props, params["language"]
+            ):
+                terms.append(Term(subject_id=subj_id, label=label, is_pref=False))
 
         return (terms, subject_ids)
 
     def _prepare_relations(self, graph, vocab):
-        self._broader_matrix = make_relation_matrix(
-            graph, vocab, SKOS.broader)
-        self._narrower_matrix = make_relation_matrix(
-            graph, vocab, SKOS.narrower)
-        self._related_matrix = make_relation_matrix(
-            graph, vocab, SKOS.related)
+        self._broader_matrix = make_relation_matrix(graph, vocab, SKOS.broader)
+        self._narrower_matrix = make_relation_matrix(graph, vocab, SKOS.narrower)
+        self._related_matrix = make_relation_matrix(graph, vocab, SKOS.related)
         self._collection_matrix = make_collection_matrix(graph, vocab)
 
     def _prepare_train_index(self, vocab, analyzer, params):
@@ -195,14 +190,12 @@ def _prepare_train_index(self, vocab, analyzer, params):
         self._prepare_relations(graph, vocab)
 
         self._vectorizer = CountVectorizer(
-            binary=True,
-            tokenizer=analyzer.tokenize_words
+            binary=True, tokenizer=analyzer.tokenize_words
         )
         label_corpus = self._vectorizer.fit_transform((t.label for t in terms))
 
         # frequency of each token used in labels - how rare each word is
-        token_freq = np.bincount(label_corpus.indices,
-                                 minlength=label_corpus.shape[1])
+        token_freq = np.bincount(label_corpus.indices, minlength=label_corpus.shape[1])
 
         self._index = TokenSetIndex()
         for term, label_matrix in zip(terms, label_corpus):
@@ -225,45 +218,43 @@ def _prepare_train_data(self, corpus, analyzer, n_jobs):
         jobs, pool_class = annif.parallel.get_pool(n_jobs)
 
         cg_args = {
-            'analyzer': analyzer,
-            'vectorizer': self._vectorizer,
-            'index': self._index
+            "analyzer": analyzer,
+            "vectorizer": self._vectorizer,
+            "index": self._index,
         }
 
-        with pool_class(jobs,
-                        initializer=MLLMCandidateGenerator.init,
-                        initargs=(cg_args,)) as pool:
-            params = ((doc.subject_set,
-                       doc.text)
-                      for doc in corpus.documents)
+        with pool_class(
+            jobs, initializer=MLLMCandidateGenerator.init, initargs=(cg_args,)
+        ) as pool:
+            params = ((doc.subject_set, doc.text) for doc in corpus.documents)
             for doc_subject_ids, candidates in pool.starmap(
-                    MLLMCandidateGenerator.generate_candidates, params, 10):
+                MLLMCandidateGenerator.generate_candidates, params, 10
+            ):
 
                 self._subj_freq.update(doc_subject_ids)
                 self._doc_freq.update([c.subject_id for c in candidates])
                 train_x.append(candidates)
-                train_y += [(c.subject_id in doc_subject_ids)
-                            for c in candidates]
+                train_y += [(c.subject_id in doc_subject_ids) for c in candidates]
 
         return (train_x, train_y)
 
     def _calculate_idf(self, subject_ids, doc_count):
         idf = collections.defaultdict(float)
         for subj_id in subject_ids:
-            idf[subj_id] = math.log((doc_count + 1) /
-                                    (self._doc_freq[subj_id] + 1)) + 1
+            idf[subj_id] = math.log((doc_count + 1) / (self._doc_freq[subj_id] + 1)) + 1
 
         return idf
 
     def _prepare_features(self, train_x, n_jobs):
-        fc_args = {'mdata': self._model_data}
+        fc_args = {"mdata": self._model_data}
         jobs, pool_class = annif.parallel.get_pool(n_jobs)
 
-        with pool_class(jobs,
-                        initializer=MLLMFeatureConverter.init,
-                        initargs=(fc_args,)) as pool:
+        with pool_class(
+            jobs, initializer=MLLMFeatureConverter.init, initargs=(fc_args,)
+        ) as pool:
             features = pool.map(
-                MLLMFeatureConverter.candidates_to_features, train_x, 10)
+                MLLMFeatureConverter.candidates_to_features, train_x, 10
+            )
 
         return features
 
@@ -272,8 +263,7 @@ def prepare_train(self, corpus, vocab, analyzer, params, n_jobs):
         subject_ids = self._prepare_train_index(vocab, analyzer, params)
 
         # convert the corpus into train data
-        train_x, train_y = self._prepare_train_data(
-            corpus, analyzer, n_jobs)
+        train_x, train_y = self._prepare_train_data(corpus, analyzer, n_jobs)
 
         # precalculate idf values for all candidate subjects
         self._idf = self._calculate_idf(subject_ids, len(train_x))
@@ -286,9 +276,11 @@ def prepare_train(self, corpus, vocab, analyzer, params, n_jobs):
     def _create_classifier(self, params):
         return BaggingClassifier(
             DecisionTreeClassifier(
-                min_samples_leaf=int(params['min_samples_leaf']),
-                max_leaf_nodes=int(params['max_leaf_nodes'])
-            ), max_samples=float(params['max_samples']))
+                min_samples_leaf=int(params["min_samples_leaf"]),
+                max_leaf_nodes=int(params["max_leaf_nodes"]),
+            ),
+            max_samples=float(params["max_samples"]),
+        )
 
     def train(self, train_x, train_y, params):
         # fit the model on the training corpus
@@ -297,14 +289,14 @@ def train(self, train_x, train_y, params):
         # sanity check: verify that the classifier has seen both classes
         if self._classifier.n_classes_ != 2:
             raise OperationFailedException(
-                "Unable to create classifier: " +
-                "Not enough positive and negative examples " +
-                "in the training data. Please check that your training " +
-                "data matches your vocabulary.")
+                "Unable to create classifier: "
+                + "Not enough positive and negative examples "
+                + "in the training data. Please check that your training "
+                + "data matches your vocabulary."
+            )
 
     def _prediction_to_list(self, scores, candidates):
-        subj_scores = [(score[1], c.subject_id)
-                       for score, c in zip(scores, candidates)]
+        subj_scores = [(score[1], c.subject_id) for score, c in zip(scores, candidates)]
         return sorted(subj_scores, reverse=True)
 
     def predict(self, candidates):
diff --git a/annif/lexical/tokenset.py b/annif/lexical/tokenset.py
index 74fa09df2..ebd23e33f 100644
--- a/annif/lexical/tokenset.py
+++ b/annif/lexical/tokenset.py
@@ -49,9 +49,10 @@ def _find_subj_tsets(self, tset):
 
         for token in tset:
             for ts in self._index[token]:
-                if tset.contains(ts) \
-                   and (ts.subject_id not in subj_tsets
-                        or not subj_tsets[ts.subject_id].is_pref):
+                if tset.contains(ts) and (
+                    ts.subject_id not in subj_tsets
+                    or not subj_tsets[ts.subject_id].is_pref
+                ):
                     subj_tsets[ts.subject_id] = ts
 
         return subj_tsets
@@ -63,11 +64,14 @@ def _find_subj_ambiguity(self, tsets):
 
         subj_ambiguity = collections.Counter()
 
-        subj_ambiguity.update([ts.subject_id
-                               for ts in tsets
-                               for other in tsets
-                               if ts != other
-                               and other.contains(ts)])
+        subj_ambiguity.update(
+            [
+                ts.subject_id
+                for ts in tsets
+                for other in tsets
+                if ts != other and other.contains(ts)
+            ]
+        )
 
         return subj_ambiguity
 
@@ -80,5 +84,6 @@ def search(self, tset):
         subj_tsets = self._find_subj_tsets(tset)
         subj_ambiguity = self._find_subj_ambiguity(subj_tsets.values())
 
-        return [(ts, subj_ambiguity[subject_id])
-                for subject_id, ts in subj_tsets.items()]
+        return [
+            (ts, subj_ambiguity[subject_id]) for subject_id, ts in subj_tsets.items()
+        ]
diff --git a/annif/lexical/util.py b/annif/lexical/util.py
index 19312b7f5..cb8402811 100644
--- a/annif/lexical/util.py
+++ b/annif/lexical/util.py
@@ -7,10 +7,12 @@
 
 
 def get_subject_labels(graph, uri, properties, language):
-    return [str(label)
-            for prop in properties
-            for label in graph.objects(URIRef(uri), prop)
-            if label.language == language]
+    return [
+        str(label)
+        for prop in properties
+        for label in graph.objects(URIRef(uri), prop)
+        if label.language == language
+    ]
 
 
 def make_relation_matrix(graph, vocab, property):
@@ -34,8 +36,7 @@ def make_collection_matrix(graph, vocab):
         if member_id is not None:
             c_members[str(coll)].append(member_id)
 
-    c_matrix = lil_matrix((len(c_members), len(vocab.subjects)),
-                          dtype=bool)
+    c_matrix = lil_matrix((len(c_members), len(vocab.subjects)), dtype=bool)
 
     # populate the matrix for collection -> subject_id
     for c_id, members in enumerate(c_members.values()):
diff --git a/annif/parallel.py b/annif/parallel.py
index 98173f564..04b914d5b 100644
--- a/annif/parallel.py
+++ b/annif/parallel.py
@@ -26,13 +26,7 @@ class ProjectSuggestMap:
     provide a mapping method that converts Document objects to suggestions.
     Intended to be used with the multiprocessing module."""
 
-    def __init__(
-            self,
-            registry,
-            project_ids,
-            backend_params,
-            limit,
-            threshold):
+    def __init__(self, registry, project_ids, backend_params, limit, threshold):
         self.registry = registry
         self.project_ids = project_ids
         self.backend_params = backend_params
@@ -45,7 +39,8 @@ def suggest(self, doc):
             project = self.registry.get_project(project_id)
             hits = project.suggest(doc.text, self.backend_params)
             filtered_hits[project_id] = hits.filter(
-                project.subjects, self.limit, self.threshold)
+                project.subjects, self.limit, self.threshold
+            )
         return (filtered_hits, doc.subject_set)
 
 
diff --git a/annif/project.py b/annif/project.py
index 22e60594d..55564143e 100644
--- a/annif/project.py
+++ b/annif/project.py
@@ -10,14 +10,19 @@
 import annif.suggestion
 import annif.backend
 from annif.datadir import DatadirMixin
-from annif.exception import AnnifException, ConfigurationException, \
-    NotSupportedException, NotInitializedException
+from annif.exception import (
+    AnnifException,
+    ConfigurationException,
+    NotSupportedException,
+    NotInitializedException,
+)
 
 logger = annif.logger
 
 
 class Access(enum.IntEnum):
     """Enumeration of access levels for projects"""
+
     private = 1
     hidden = 2
     public = 3
@@ -35,44 +40,45 @@ class AnnifProject(DatadirMixin):
     initialized = False
 
     # default values for configuration settings
-    DEFAULT_ACCESS = 'public'
+    DEFAULT_ACCESS = "public"
 
     def __init__(self, project_id, config, datadir, registry):
-        DatadirMixin.__init__(self, datadir, 'projects', project_id)
+        DatadirMixin.__init__(self, datadir, "projects", project_id)
         self.project_id = project_id
-        self.name = config.get('name', project_id)
-        self.language = config['language']
-        self.analyzer_spec = config.get('analyzer', None)
-        self.transform_spec = config.get('transform', 'pass')
-        self.vocab_spec = config.get('vocab', None)
+        self.name = config.get("name", project_id)
+        self.language = config["language"]
+        self.analyzer_spec = config.get("analyzer", None)
+        self.transform_spec = config.get("transform", "pass")
+        self.vocab_spec = config.get("vocab", None)
         self.config = config
         self._base_datadir = datadir
         self.registry = registry
         self._init_access()
 
     def _init_access(self):
-        access = self.config.get('access', self.DEFAULT_ACCESS)
+        access = self.config.get("access", self.DEFAULT_ACCESS)
         try:
             self.access = getattr(Access, access)
         except AttributeError:
             raise ConfigurationException(
                 "'{}' is not a valid access setting".format(access),
-                project_id=self.project_id)
+                project_id=self.project_id,
+            )
 
     def _initialize_analyzer(self):
         if not self.analyzer_spec:
             return  # not configured, so assume it's not needed
         analyzer = self.analyzer
-        logger.debug("Project '%s': initialized analyzer: %s",
-                     self.project_id,
-                     str(analyzer))
+        logger.debug(
+            "Project '%s': initialized analyzer: %s", self.project_id, str(analyzer)
+        )
 
     def _initialize_subjects(self):
         try:
             subjects = self.subjects
-            logger.debug("Project '%s': initialized subjects: %s",
-                         self.project_id,
-                         str(subjects))
+            logger.debug(
+                "Project '%s': initialized subjects: %s", self.project_id, str(subjects)
+            )
         except AnnifException as err:
             logger.warning(err.format_message())
 
@@ -107,54 +113,57 @@ def _suggest_with_backend(self, text, backend_params):
             backend_params = {}
         beparams = backend_params.get(self.backend.backend_id, {})
         hits = self.backend.suggest(text, beparams)
-        logger.debug(
-            'Got %d hits from backend %s',
-            len(hits), self.backend.backend_id)
+        logger.debug("Got %d hits from backend %s", len(hits), self.backend.backend_id)
         return hits
 
     @property
     def analyzer(self):
         if self._analyzer is None:
             if self.analyzer_spec:
-                self._analyzer = annif.analyzer.get_analyzer(
-                    self.analyzer_spec)
+                self._analyzer = annif.analyzer.get_analyzer(self.analyzer_spec)
             else:
                 raise ConfigurationException(
-                    "analyzer setting is missing", project_id=self.project_id)
+                    "analyzer setting is missing", project_id=self.project_id
+                )
         return self._analyzer
 
     @property
     def transform(self):
         if self._transform is None:
             self._transform = annif.transform.get_transform(
-                self.transform_spec, project=self)
+                self.transform_spec, project=self
+            )
         return self._transform
 
     @property
     def backend(self):
         if self._backend is None:
-            if 'backend' not in self.config:
+            if "backend" not in self.config:
                 raise ConfigurationException(
-                    "backend setting is missing", project_id=self.project_id)
-            backend_id = self.config['backend']
+                    "backend setting is missing", project_id=self.project_id
+                )
+            backend_id = self.config["backend"]
             try:
                 backend_class = annif.backend.get_backend(backend_id)
                 self._backend = backend_class(
-                    backend_id, config_params=self.config,
-                    project=self)
+                    backend_id, config_params=self.config, project=self
+                )
             except ValueError:
                 logger.warning(
                     "Could not create backend %s, "
                     "make sure you've installed optional dependencies",
-                    backend_id)
+                    backend_id,
+                )
         return self._backend
 
     def _initialize_vocab(self):
         if self.vocab_spec is None:
-            raise ConfigurationException("vocab setting is missing",
-                                         project_id=self.project_id)
+            raise ConfigurationException(
+                "vocab setting is missing", project_id=self.project_id
+            )
         self._vocab, self._vocab_lang = self.registry.get_vocab(
-            self.vocab_spec, self.language)
+            self.vocab_spec, self.language
+        )
 
     @property
     def vocab(self):
@@ -183,30 +192,31 @@ def _get_info(self, key):
 
     @property
     def is_trained(self):
-        return self._get_info('is_trained')
+        return self._get_info("is_trained")
 
     @property
     def modification_time(self):
-        return self._get_info('modification_time')
+        return self._get_info("modification_time")
 
     def suggest(self, text, backend_params=None):
         """Suggest subjects the given text by passing it to the backend. Returns a
         list of SubjectSuggestion objects ordered by decreasing score."""
         if not self.is_trained:
             if self.is_trained is None:
-                logger.warning('Could not get train state information.')
+                logger.warning("Could not get train state information.")
             else:
-                raise NotInitializedException('Project is not trained.')
-        logger.debug('Suggesting subjects for text "%s..." (len=%d)',
-                     text[:20], len(text))
+                raise NotInitializedException("Project is not trained.")
+        logger.debug(
+            'Suggesting subjects for text "%s..." (len=%d)', text[:20], len(text)
+        )
         text = self.transform.transform_text(text)
         hits = self._suggest_with_backend(text, backend_params)
-        logger.debug('%d hits from backend', len(hits))
+        logger.debug("%d hits from backend", len(hits))
         return hits
 
     def train(self, corpus, backend_params=None, jobs=0):
         """train the project using documents from a metadata source"""
-        if corpus != 'cached':
+        if corpus != "cached":
             corpus = self.transform.transform_corpus(corpus)
         if backend_params is None:
             backend_params = {}
@@ -219,44 +229,43 @@ def learn(self, corpus, backend_params=None):
             backend_params = {}
         beparams = backend_params.get(self.backend.backend_id, {})
         corpus = self.transform.transform_corpus(corpus)
-        if isinstance(
-                self.backend,
-                annif.backend.backend.AnnifLearningBackend):
+        if isinstance(self.backend, annif.backend.backend.AnnifLearningBackend):
             self.backend.learn(corpus, beparams)
         else:
-            raise NotSupportedException("Learning not supported by backend",
-                                        project_id=self.project_id)
+            raise NotSupportedException(
+                "Learning not supported by backend", project_id=self.project_id
+            )
 
     def hyperopt(self, corpus, trials, jobs, metric, results_file):
         """optimize the hyperparameters of the project using a validation
         corpus against a given metric"""
-        if isinstance(
-                self.backend,
-                annif.backend.hyperopt.AnnifHyperoptBackend):
+        if isinstance(self.backend, annif.backend.hyperopt.AnnifHyperoptBackend):
             optimizer = self.backend.get_hp_optimizer(corpus, metric)
             return optimizer.optimize(trials, jobs, results_file)
 
         raise NotSupportedException(
-            "Hyperparameter optimization not supported "
-            "by backend", project_id=self.project_id)
+            "Hyperparameter optimization not supported " "by backend",
+            project_id=self.project_id,
+        )
 
     def dump(self):
         """return this project as a dict"""
-        return {'project_id': self.project_id,
-                'name': self.name,
-                'language': self.language,
-                'backend': {'backend_id': self.config.get('backend')},
-                'is_trained': self.is_trained,
-                'modification_time': self.modification_time
-                }
+        return {
+            "project_id": self.project_id,
+            "name": self.name,
+            "language": self.language,
+            "backend": {"backend_id": self.config.get("backend")},
+            "is_trained": self.is_trained,
+            "modification_time": self.modification_time,
+        }
 
     def remove_model_data(self):
         """remove the data of this project"""
         datadir_path = self._datadir_path
         if os.path.isdir(datadir_path):
             rmtree(datadir_path)
-            logger.info('Removed model data for project {}.'
-                        .format(self.project_id))
+            logger.info("Removed model data for project {}.".format(self.project_id))
         else:
-            logger.warning('No model data to remove for project {}.'
-                           .format(self.project_id))
+            logger.warning(
+                "No model data to remove for project {}.".format(self.project_id)
+            )
diff --git a/annif/registry.py b/annif/registry.py
index 57c77a94c..70cfc520f 100644
--- a/annif/registry.py
+++ b/annif/registry.py
@@ -29,8 +29,7 @@ class AnnifRegistry:
     def __init__(self, projects_config_path, datadir, init_projects):
         self._rid = id(self)
         self._datadir = datadir
-        self._projects[self._rid] = \
-            self._create_projects(projects_config_path)
+        self._projects[self._rid] = self._create_projects(projects_config_path)
         self._vocabs[self._rid] = {}
         if init_projects:
             for project in self._projects[self._rid].values():
@@ -47,10 +46,9 @@ def _create_projects(self, projects_config_path):
         # create AnnifProject objects from the configuration file
         projects = collections.OrderedDict()
         for project_id in config.project_ids:
-            projects[project_id] = AnnifProject(project_id,
-                                                config[project_id],
-                                                self._datadir,
-                                                self)
+            projects[project_id] = AnnifProject(
+                project_id, config[project_id], self._datadir, self
+            )
         return projects
 
     def get_projects(self, min_access=Access.private):
@@ -58,9 +56,11 @@ def get_projects(self, min_access=Access.private):
         AnnifProject. The min_access parameter may be used to set the minimum
         access level required for the returned projects."""
 
-        return {project_id: project
-                for project_id, project in self._projects[self._rid].items()
-                if project.access >= min_access}
+        return {
+            project_id: project
+            for project_id, project in self._projects[self._rid].items()
+            if project.access >= min_access
+        }
 
     def get_project(self, project_id, min_access=Access.private):
         """return the definition of a single Project by project_id"""
@@ -76,10 +76,9 @@ def get_vocab(self, vocab_spec, default_language):
         vocab_spec. If no language information is specified, use the given
         default language."""
 
-        match = re.match(r'([\w-]+)(\((.*)\))?$', vocab_spec)
+        match = re.match(r"([\w-]+)(\((.*)\))?$", vocab_spec)
         if match is None:
-            raise ValueError(
-                f"Invalid vocabulary specification: {vocab_spec}")
+            raise ValueError(f"Invalid vocabulary specification: {vocab_spec}")
         vocab_id = match.group(1)
         posargs, kwargs = parse_args(match.group(3))
         language = posargs[0] if posargs else default_language
@@ -87,23 +86,23 @@ def get_vocab(self, vocab_spec, default_language):
 
         if vocab_key not in self._vocabs[self._rid]:
             self._vocabs[self._rid][vocab_key] = AnnifVocabulary(
-                vocab_id, self._datadir)
+                vocab_id, self._datadir
+            )
         return self._vocabs[self._rid][vocab_key], language
 
 
 def initialize_projects(app):
-    projects_config_path = app.config['PROJECTS_CONFIG_PATH']
-    datadir = app.config['DATADIR']
-    init_projects = app.config['INITIALIZE_PROJECTS']
-    app.annif_registry = AnnifRegistry(projects_config_path, datadir,
-                                       init_projects)
+    projects_config_path = app.config["PROJECTS_CONFIG_PATH"]
+    datadir = app.config["DATADIR"]
+    init_projects = app.config["INITIALIZE_PROJECTS"]
+    app.annif_registry = AnnifRegistry(projects_config_path, datadir, init_projects)
 
 
 def get_projects(min_access=Access.private):
     """Return the available projects as a dict of project_id ->
     AnnifProject. The min_access parameter may be used to set the minimum
     access level required for the returned projects."""
-    if not hasattr(current_app, 'annif_registry'):
+    if not hasattr(current_app, "annif_registry"):
         initialize_projects(current_app)
 
     return current_app.annif_registry.get_projects(min_access)
diff --git a/annif/rest.py b/annif/rest.py
index df2c45b7a..540f82e7d 100644
--- a/annif/rest.py
+++ b/annif/rest.py
@@ -14,8 +14,9 @@ def project_not_found_error(project_id):
 
     return connexion.problem(
         status=404,
-        title='Project not found',
-        detail="Project '{}' not found".format(project_id))
+        title="Project not found",
+        detail="Project '{}' not found".format(project_id),
+    )
 
 
 def server_error(err):
@@ -23,26 +24,26 @@ def server_error(err):
     or backend problem)"""
 
     return connexion.problem(
-        status=503,
-        title='Service unavailable',
-        detail=err.format_message())
+        status=503, title="Service unavailable", detail=err.format_message()
+    )
 
 
 def list_projects():
     """return a dict with projects formatted according to Swagger spec"""
 
     return {
-        'projects': [
-            proj.dump() for proj in annif.registry.get_projects(
-                min_access=Access.public).values()]}
+        "projects": [
+            proj.dump()
+            for proj in annif.registry.get_projects(min_access=Access.public).values()
+        ]
+    }
 
 
 def show_project(project_id):
     """return a single project formatted according to Swagger spec"""
 
     try:
-        project = annif.registry.get_project(
-            project_id, min_access=Access.hidden)
+        project = annif.registry.get_project(project_id, min_access=Access.hidden)
     except ValueError:
         return project_not_found_error(project_id)
     return project.dump()
@@ -51,10 +52,10 @@ def show_project(project_id):
 def _suggestion_to_dict(suggestion, subject_index, language):
     subject = subject_index[suggestion.subject_id]
     return {
-        'uri': subject.uri,
-        'label': subject.labels[language],
-        'notation': subject.notation,
-        'score': suggestion.score
+        "uri": subject.uri,
+        "label": subject.labels[language],
+        "notation": subject.notation,
+        "score": suggestion.score,
     }
 
 
@@ -63,8 +64,7 @@ def suggest(project_id, text, limit, threshold, language=None):
     formatted according to Swagger spec"""
 
     try:
-        project = annif.registry.get_project(
-            project_id, min_access=Access.hidden)
+        project = annif.registry.get_project(project_id, min_access=Access.hidden)
     except ValueError:
         return project_not_found_error(project_id)
 
@@ -76,8 +76,9 @@ def suggest(project_id, text, limit, threshold, language=None):
     if lang not in project.vocab.languages:
         return connexion.problem(
             status=400,
-            title='Bad Request',
-            detail=f'language "{lang}" not supported by vocabulary')
+            title="Bad Request",
+            detail=f'language "{lang}" not supported by vocabulary',
+        )
 
     try:
         hit_filter = SuggestionFilter(project.subjects, limit, threshold)
@@ -86,17 +87,22 @@ def suggest(project_id, text, limit, threshold, language=None):
         return server_error(err)
 
     hits = hit_filter(result).as_list()
-    return {'results': [_suggestion_to_dict(hit, project.subjects, lang)
-                        for hit in hits]}
+    return {
+        "results": [_suggestion_to_dict(hit, project.subjects, lang) for hit in hits]
+    }
 
 
 def _documents_to_corpus(documents, subject_index):
-    corpus = [Document(text=d['text'],
-                       subject_set=SubjectSet(
-                           [subject_index.by_uri(subj['uri'])
-                            for subj in d['subjects']]))
-              for d in documents
-              if 'text' in d and 'subjects' in d]
+    corpus = [
+        Document(
+            text=d["text"],
+            subject_set=SubjectSet(
+                [subject_index.by_uri(subj["uri"]) for subj in d["subjects"]]
+            ),
+        )
+        for d in documents
+        if "text" in d and "subjects" in d
+    ]
     return DocumentList(corpus)
 
 
@@ -104,8 +110,7 @@ def learn(project_id, documents):
     """learn from documents and return an empty 204 response if succesful"""
 
     try:
-        project = annif.registry.get_project(
-            project_id, min_access=Access.hidden)
+        project = annif.registry.get_project(project_id, min_access=Access.hidden)
     except ValueError:
         return project_not_found_error(project_id)
 
diff --git a/annif/suggestion.py b/annif/suggestion.py
index 74f991b6f..10f2cfd8b 100644
--- a/annif/suggestion.py
+++ b/annif/suggestion.py
@@ -6,10 +6,10 @@
 import numpy as np
 
 
-SubjectSuggestion = collections.namedtuple(
-    'SubjectSuggestion', 'subject_id score')
+SubjectSuggestion = collections.namedtuple("SubjectSuggestion", "subject_id score")
 WeightedSuggestion = collections.namedtuple(
-    'WeightedSuggestion', 'hits weight subjects')
+    "WeightedSuggestion", "hits weight subjects"
+)
 
 
 class SuggestionFilter:
@@ -22,9 +22,8 @@ def __init__(self, subject_index, limit=None, threshold=0.0):
 
     def __call__(self, orighits):
         return LazySuggestionResult(
-            lambda: orighits.filter(self._subject_index,
-                                    self._limit,
-                                    self._threshold))
+            lambda: orighits.filter(self._subject_index, self._limit, self._threshold)
+        )
 
 
 class SuggestionResult(metaclass=abc.ABCMeta):
@@ -104,10 +103,7 @@ def _vector_to_list_suggestion(self):
             score = self._vector[subject_id]
             if score <= 0.0:
                 break  # we can skip the remaining ones
-            hits.append(
-                SubjectSuggestion(
-                    subject_id=subject_id,
-                    score=float(score)))
+            hits.append(SubjectSuggestion(subject_id=subject_id, score=float(score)))
         return ListSuggestionResult(hits)
 
     @property
@@ -128,14 +124,15 @@ def as_vector(self, size, destination=None):
         return self._vector
 
     def filter(self, subject_index, limit=None, threshold=0.0):
-        mask = (self._vector > threshold)
+        mask = self._vector > threshold
         deprecated_ids = subject_index.deprecated_ids()
         if limit is not None:
             limit_mask = np.zeros_like(self._vector, dtype=bool)
             deprecated_set = set(deprecated_ids)
             top_k_subjects = itertools.islice(
-                                (subj for subj in self.subject_order
-                                 if subj not in deprecated_set), limit)
+                (subj for subj in self.subject_order if subj not in deprecated_set),
+                limit,
+            )
             limit_mask[list(top_k_subjects)] = True
             mask = mask & limit_mask
         else:
@@ -153,9 +150,7 @@ class ListSuggestionResult(SuggestionResult):
     """SuggestionResult implementation based primarily on lists of hits."""
 
     def __init__(self, hits):
-        self._list = [self._enforce_score_range(hit)
-                      for hit in hits
-                      if hit.score > 0.0]
+        self._list = [self._enforce_score_range(hit) for hit in hits if hit.score > 0.0]
         self._vector = None
 
     @staticmethod
@@ -183,9 +178,11 @@ def as_vector(self, size, destination=None):
 
     def filter(self, subject_index, limit=None, threshold=0.0):
         hits = sorted(self._list, key=lambda hit: hit.score, reverse=True)
-        filtered_hits = [hit for hit in hits
-                         if hit.score >= threshold and hit.score > 0.0 and
-                         hit.subject_id is not None]
+        filtered_hits = [
+            hit
+            for hit in hits
+            if hit.score >= threshold and hit.score > 0.0 and hit.subject_id is not None
+        ]
         if limit is not None:
             filtered_hits = filtered_hits[:limit]
         return ListSuggestionResult(filtered_hits)
diff --git a/annif/transform/__init__.py b/annif/transform/__init__.py
index 61b9a6c59..76a61461f 100644
--- a/annif/transform/__init__.py
+++ b/annif/transform/__init__.py
@@ -15,9 +15,9 @@ def parse_specs(transform_specs):
 
     parsed = []
     # Split by commas not inside parentheses
-    parts = re.split(r',\s*(?![^()]*\))', transform_specs)
+    parts = re.split(r",\s*(?![^()]*\))", transform_specs)
     for part in parts:
-        match = re.match(r'(\w+)(\((.*)\))?', part)
+        match = re.match(r"(\w+)(\((.*)\))?", part)
         if match is None:
             continue
         transform = match.group(1)
@@ -40,12 +40,13 @@ def get_transform(transform_specs, project):
 
 _transforms = {
     transform.IdentityTransform.name: transform.IdentityTransform,
-    inputlimiter.InputLimiter.name: inputlimiter.InputLimiter}
+    inputlimiter.InputLimiter.name: inputlimiter.InputLimiter,
+}
 
 # Optional transforms
 try:
     from . import langfilter
+
     _transforms.update({langfilter.LangFilter.name: langfilter.LangFilter})
 except ImportError:
-    annif.logger.debug(
-        "pycld3 not available, not enabling filter_language transform")
+    annif.logger.debug("pycld3 not available, not enabling filter_language transform")
diff --git a/annif/transform/inputlimiter.py b/annif/transform/inputlimiter.py
index 7802972db..0a4748db9 100644
--- a/annif/transform/inputlimiter.py
+++ b/annif/transform/inputlimiter.py
@@ -7,7 +7,7 @@
 
 class InputLimiter(transform.BaseTransform):
 
-    name = 'limit'
+    name = "limit"
 
     def __init__(self, project, input_limit):
         super().__init__(project)
@@ -15,10 +15,11 @@ def __init__(self, project, input_limit):
         self._validate_value(self.input_limit)
 
     def transform_fn(self, text):
-        return text[:self.input_limit]
+        return text[: self.input_limit]
 
     def _validate_value(self, input_limit):
         if input_limit < 0:
             raise ConfigurationException(
-                'input_limit in limit_input transform cannot be negative',
-                project_id=self.project.project_id)
+                "input_limit in limit_input transform cannot be negative",
+                project_id=self.project.project_id,
+            )
diff --git a/annif/transform/langfilter.py b/annif/transform/langfilter.py
index f63f6f97a..266142fd7 100644
--- a/annif/transform/langfilter.py
+++ b/annif/transform/langfilter.py
@@ -10,7 +10,7 @@
 
 class LangFilter(transform.BaseTransform):
 
-    name = 'filter_lang'
+    name = "filter_lang"
 
     def __init__(self, project, text_min_length=500, sentence_min_length=50):
         super().__init__(project)
@@ -39,4 +39,4 @@ def transform_fn(self, text):
             detected_lang = self._detect_language(sent)
             if detected_lang == self.project.language or detected_lang is None:
                 retained_sentences.append(sent)
-        return ' '.join(retained_sentences)
+        return " ".join(retained_sentences)
diff --git a/annif/transform/transform.py b/annif/transform/transform.py
index 924461881..f91549e67 100644
--- a/annif/transform/transform.py
+++ b/annif/transform/transform.py
@@ -23,13 +23,13 @@ def transform_fn(self, text):
 class IdentityTransform(BaseTransform):
     """Transform that does not modify text but simply passes it through."""
 
-    name = 'pass'
+    name = "pass"
 
     def transform_fn(self, text):
         return text
 
 
-class TransformChain():
+class TransformChain:
     """Class instantiating and holding the transformation objects performing
     the actual text transformation."""
 
@@ -41,13 +41,13 @@ def _init_transforms(self, transform_classes, args):
         transforms = []
         for trans, (posargs, kwargs) in zip(transform_classes, args):
             try:
-                transforms.append(
-                    trans(self.project, *posargs, **kwargs))
+                transforms.append(trans(self.project, *posargs, **kwargs))
             except (ValueError, TypeError):
                 raise ConfigurationException(
                     f"Invalid arguments to {trans.name} transform: "
                     f"{posargs}, {kwargs})",
-                    project_id=self.project.project_id)
+                    project_id=self.project.project_id,
+                )
         return transforms
 
     def transform_text(self, text):
diff --git a/annif/util.py b/annif/util.py
index c05fc3892..9e2ffc096 100644
--- a/annif/util.py
+++ b/annif/util.py
@@ -16,23 +16,22 @@ def atomic_save(obj, dirname, filename, method=None):
     final name."""
 
     prefix, suffix = os.path.splitext(filename)
-    tempfd, tempfilename = tempfile.mkstemp(
-        prefix=prefix, suffix=suffix, dir=dirname)
+    tempfd, tempfilename = tempfile.mkstemp(prefix=prefix, suffix=suffix, dir=dirname)
     os.close(tempfd)
-    logger.debug('saving %s to temporary file %s', str(obj)[:90], tempfilename)
+    logger.debug("saving %s to temporary file %s", str(obj)[:90], tempfilename)
     if method is not None:
         method(obj, tempfilename)
     else:
         obj.save(tempfilename)
-    for fn in glob.glob(tempfilename + '*'):
+    for fn in glob.glob(tempfilename + "*"):
         newname = fn.replace(tempfilename, os.path.join(dirname, filename))
-        logger.debug('renaming temporary file %s to %s', fn, newname)
+        logger.debug("renaming temporary file %s to %s", fn, newname)
         os.rename(fn, newname)
 
 
 def cleanup_uri(uri):
     """remove angle brackets from a URI, if any"""
-    if uri.startswith('<') and uri.endswith('>'):
+    if uri.startswith("<") and uri.endswith(">"):
         return uri[1:-1]
     return uri
 
@@ -54,8 +53,8 @@ def parse_sources(sourcedef):
 
     sources = []
     totalweight = 0.0
-    for srcdef in sourcedef.strip().split(','):
-        srcval = srcdef.strip().split(':')
+    for srcdef in sourcedef.strip().split(","):
+        srcval = srcdef.strip().split(":")
         src_id = srcval[0]
         if len(srcval) > 1:
             weight = float(srcval[1])
@@ -74,9 +73,9 @@ def parse_args(param_string):
         return [], {}
     posargs = []
     kwargs = {}
-    param_strings = param_string.split(',')
+    param_strings = param_string.split(",")
     for p_string in param_strings:
-        parts = p_string.split('=')
+        parts = p_string.split("=")
         if len(parts) == 1:
             posargs.append(p_string)
         elif len(parts) == 2:
@@ -89,7 +88,7 @@ def boolean(val):
     True values are '1', 'yes', 'true', and 'on' (case insensitive), everything
     else is False."""
 
-    return str(val).lower() in ('1', 'yes', 'true', 'on')
+    return str(val).lower() in ("1", "yes", "true", "on")
 
 
 def identity(x):
@@ -99,4 +98,4 @@ def identity(x):
 
 def metric_code(metric):
     """Convert a human-readable metric name into an alphanumeric string"""
-    return metric.translate(metric.maketrans(' ', '_', '()'))
+    return metric.translate(metric.maketrans(" ", "_", "()"))
diff --git a/annif/views.py b/annif/views.py
index 2aeaae293..44b0028c7 100644
--- a/annif/views.py
+++ b/annif/views.py
@@ -1,8 +1,8 @@
 from flask import Blueprint, render_template
 
-bp = Blueprint('app', __name__)
+bp = Blueprint("app", __name__)
 
 
-@bp.route('/')
+@bp.route("/")
 def home():
-    return render_template('home.html')
+    return render_template("home.html")
diff --git a/annif/vocab.py b/annif/vocab.py
index 769d67431..fd9f0c774 100644
--- a/annif/vocab.py
+++ b/annif/vocab.py
@@ -23,15 +23,14 @@ class AnnifVocabulary(DatadirMixin):
     INDEX_FILENAME_CSV = "subjects.csv"
 
     def __init__(self, vocab_id, datadir):
-        DatadirMixin.__init__(self, datadir, 'vocabs', vocab_id)
+        DatadirMixin.__init__(self, datadir, "vocabs", vocab_id)
         self.vocab_id = vocab_id
         self._skos_vocab = None
 
     def _create_subject_index(self, subject_corpus):
         subjects = annif.corpus.SubjectIndex()
         subjects.load_subjects(subject_corpus)
-        annif.util.atomic_save(subjects, self.datadir,
-                               self.INDEX_FILENAME_CSV)
+        annif.util.atomic_save(subjects, self.datadir, self.INDEX_FILENAME_CSV)
         return subjects
 
     def _update_subject_index(self, subject_corpus):
@@ -42,18 +41,16 @@ def _update_subject_index(self, subject_corpus):
 
         for old_subject in old_subjects:
             if new_subjects.contains_uri(old_subject.uri):
-                new_subject = new_subjects[new_subjects.by_uri(
-                    old_subject.uri)]
+                new_subject = new_subjects[new_subjects.by_uri(old_subject.uri)]
             else:  # subject removed from new corpus
-                new_subject = annif.corpus.Subject(uri=old_subject.uri,
-                                                   labels=None,
-                                                   notation=None)
+                new_subject = annif.corpus.Subject(
+                    uri=old_subject.uri, labels=None, notation=None
+                )
             updated_subjects.append(new_subject)
         for new_subject in new_subjects:
             if not old_subjects.contains_uri(new_subject.uri):
                 updated_subjects.append(new_subject)
-        annif.util.atomic_save(updated_subjects, self.datadir,
-                               self.INDEX_FILENAME_CSV)
+        annif.util.atomic_save(updated_subjects, self.datadir, self.INDEX_FILENAME_CSV)
         return updated_subjects
 
     @property
@@ -61,11 +58,10 @@ def subjects(self):
         if self._subjects is None:
             path = os.path.join(self.datadir, self.INDEX_FILENAME_CSV)
             if os.path.exists(path):
-                logger.debug('loading subjects from %s', path)
+                logger.debug("loading subjects from %s", path)
                 self._subjects = annif.corpus.SubjectIndex.load(path)
             else:
-                raise NotInitializedException(
-                    "subject file {} not found".format(path))
+                raise NotInitializedException("subject file {} not found".format(path))
         return self._subjects
 
     @property
@@ -77,25 +73,25 @@ def skos(self):
         # attempt to load graph from dump file
         dumppath = os.path.join(self.datadir, self.INDEX_FILENAME_DUMP)
         if os.path.exists(dumppath):
-            logger.debug(f'loading graph dump from {dumppath}')
+            logger.debug(f"loading graph dump from {dumppath}")
             try:
                 self._skos_vocab = annif.corpus.SubjectFileSKOS(dumppath)
             except ModuleNotFoundError:
                 # Probably dump has been saved using a different rdflib version
-                logger.debug('could not load graph dump, using turtle file')
+                logger.debug("could not load graph dump, using turtle file")
             else:
                 return self._skos_vocab
 
         # graph dump file not found - parse ttl file instead
         path = os.path.join(self.datadir, self.INDEX_FILENAME_TTL)
         if os.path.exists(path):
-            logger.debug(f'loading graph from {path}')
+            logger.debug(f"loading graph from {path}")
             self._skos_vocab = annif.corpus.SubjectFileSKOS(path)
             # store the dump file so we can use it next time
             self._skos_vocab.save_skos(path)
             return self._skos_vocab
 
-        raise NotInitializedException(f'graph file {path} not found')
+        raise NotInitializedException(f"graph file {path} not found")
 
     def __len__(self):
         return len(self.subjects)
@@ -110,15 +106,16 @@ def load_vocabulary(self, subject_corpus, force=False):
         use. If force=True, replace the existing subject index completely."""
 
         if not force and os.path.exists(
-                os.path.join(self.datadir, self.INDEX_FILENAME_CSV)):
-            logger.info('updating existing subject index')
+            os.path.join(self.datadir, self.INDEX_FILENAME_CSV)
+        ):
+            logger.info("updating existing subject index")
             self._subjects = self._update_subject_index(subject_corpus)
         else:
-            logger.info('creating subject index')
+            logger.info("creating subject index")
             self._subjects = self._create_subject_index(subject_corpus)
 
         skosfile = os.path.join(self.datadir, self.INDEX_FILENAME_TTL)
-        logger.info(f'saving vocabulary into SKOS file {skosfile}')
+        logger.info(f"saving vocabulary into SKOS file {skosfile}")
         subject_corpus.save_skos(skosfile)
 
     def as_graph(self):
diff --git a/autopep8.sh b/autopep8.sh
deleted file mode 100755
index eef99e49b..000000000
--- a/autopep8.sh
+++ /dev/null
@@ -1,10 +0,0 @@
-#!/bin/bash
-
-if [ "$1" = "-i" ]; then
-	echo "Running autopep8 in in-place mode."
-	exec autopep8 -i -j0 -a -a -r annif tests docs *.py
-else
-	echo "Running autopep8 in diff mode. Use -i for in-place mode."
-	echo ""
-	exec autopep8 -d -a -a -r annif tests docs *.py
-fi
diff --git a/docs/conf.py b/docs/conf.py
index f9e1cb9c1..17183ac12 100644
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -13,48 +13,45 @@
 import os
 import re
 import sys
-sys.path.insert(0, os.path.abspath('..'))
+
+sys.path.insert(0, os.path.abspath(".."))
 
 
 # -- Project information -----------------------------------------------------
 
-project = 'annif'
-copyright = '2017, University Of Helsinki (The National Library Of Finland)'
-author = 'Osma Suominen'
+project = "annif"
+copyright = "2017, University Of Helsinki (The National Library Of Finland)"
+author = "Osma Suominen"
 
 # Get version number from GitHub tag
-release = re.sub(
-    '^v',
-    '',
-    os.popen('git describe --tags').read().strip()
-)
+release = re.sub("^v", "", os.popen("git describe --tags").read().strip())
 # The short X.Y version.
 version = release
 
 
 # -- General configuration ---------------------------------------------------
 
-master_doc = 'index'
+master_doc = "index"
 
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
 extensions = [
-    'sphinx.ext.autodoc',
-    'sphinxcontrib.apidoc',
-    'sphinx_click',
+    "sphinx.ext.autodoc",
+    "sphinxcontrib.apidoc",
+    "sphinx_click",
 ]
 
 # Add any paths that contain templates here, relative to this directory.
-templates_path = ['_templates']
+templates_path = ["_templates"]
 
 # List of patterns, relative to source directory, that match files and
 # directories to ignore when looking for source files.
 # This pattern also affects html_static_path and html_extra_path.
-exclude_patterns = ['_build', 'Thumbs.db', '.DS_Store', 'venv']
+exclude_patterns = ["_build", "Thumbs.db", ".DS_Store", "venv"]
 
-apidoc_module_dir = '../annif'
-apidoc_output_dir = 'source'
+apidoc_module_dir = "../annif"
+apidoc_output_dir = "source"
 apidoc_separate_modules = False
 
 # -- Options for HTML output -------------------------------------------------
@@ -62,13 +59,13 @@
 # The theme to use for HTML and HTML Help pages.  See the documentation for
 # a list of builtin themes.
 #
-html_theme = 'sphinx_rtd_theme'
-html_logo = 'source/annif-RGB-white.png'
+html_theme = "sphinx_rtd_theme"
+html_logo = "source/annif-RGB-white.png"
 html_theme_options = {
-    'logo_only': True,
+    "logo_only": True,
 }
 
 # Add any paths that contain custom static files (such as style sheets) here,
 # relative to this directory. They are copied after the builtin static files,
 # so a file named "default.css" will overwrite the builtin "default.css".
-html_static_path = ['']
+html_static_path = [""]
diff --git a/pyproject.toml b/pyproject.toml
index e6bb92f1e..d91f1e2c8 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -71,7 +71,7 @@ pytest-flask = "*"
 pytest-flake8 = "*"
 flake8 = "4.*"
 bumpversion = "*"
-autopep8 = "*"
+black = "*"
 
 [tool.poetry.extras]
 fasttext = ["fasttext-wheel"]
diff --git a/setup.cfg b/setup.cfg
index 51edbf3e9..8497fbe58 100644
--- a/setup.cfg
+++ b/setup.cfg
@@ -19,3 +19,7 @@ values =
 
 [aliases]
 test = pytest
+
+[flake8]
+max-line-length = 88
+ignore = E203 W503


Test patch:
diff --git a/tests/conftest.py b/tests/conftest.py
index 2d5251cb1..35810191b 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -12,26 +12,22 @@
 import annif.registry
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def app():
     # make sure the dummy vocab is in place because many tests depend on it
-    subjfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'dummy-subjects.csv')
-    app = annif.create_app(config_name='annif.default_config.TestingConfig')
+    subjfile = os.path.join(os.path.dirname(__file__), "corpora", "dummy-subjects.csv")
+    app = annif.create_app(config_name="annif.default_config.TestingConfig")
     with app.app_context():
-        project = annif.registry.get_project('dummy-en')
+        project = annif.registry.get_project("dummy-en")
         # the vocab is needed for both English and Finnish language projects
         vocab = annif.corpus.SubjectFileCSV(subjfile)
         project.vocab.load_vocabulary(vocab)
     return app
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def app_with_initialize():
-    app = annif.create_app(
-        config_name='annif.default_config.TestingInitializeConfig')
+    app = annif.create_app(config_name="annif.default_config.TestingInitializeConfig")
     return app
 
 
@@ -41,99 +37,91 @@ def app_client(app):
         yield app_client
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def registry(app):
     with app.app_context():
         return app.annif_registry
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def datadir(tmpdir_factory):
-    return tmpdir_factory.mktemp('data')
+    return tmpdir_factory.mktemp("data")
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def testdatadir(app):
     """a fixture to access the tests/data directory as a py.path.local
-     object"""
+    object"""
     with app.app_context():
-        dir = py.path.local(app.config['DATADIR'])
+        dir = py.path.local(app.config["DATADIR"])
     # clean up previous state of datadir
-    shutil.rmtree(os.path.join(str(dir), 'projects'), ignore_errors=True)
+    shutil.rmtree(os.path.join(str(dir), "projects"), ignore_errors=True)
     return dir
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def subject_file():
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'subjects.tsv')
-    return annif.corpus.SubjectFileTSV(docfile, 'fi')
+        os.path.dirname(__file__), "corpora", "archaeology", "subjects.tsv"
+    )
+    return annif.corpus.SubjectFileTSV(docfile, "fi")
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def dummy_subject_index(testdatadir):
     """a fixture to access the subject index of the dummy vocabulary"""
-    vocab = annif.vocab.AnnifVocabulary('dummy', testdatadir)
+    vocab = annif.vocab.AnnifVocabulary("dummy", testdatadir)
     return vocab.subjects
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def vocabulary(datadir):
-    vocab = annif.vocab.AnnifVocabulary('my-vocab', datadir)
+    vocab = annif.vocab.AnnifVocabulary("my-vocab", datadir)
     subjfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'yso-archaeology.ttl')
+        os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
+    )
     subjects = annif.corpus.SubjectFileSKOS(subjfile)
     vocab.load_vocabulary(subjects)
     return vocab
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def subject_index(vocabulary):
     return vocabulary.subjects
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def document_corpus(subject_index):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
     doc_corpus = annif.corpus.DocumentFile(docfile, subject_index)
     return doc_corpus
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def fulltext_corpus(subject_index):
     ftdir = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'fulltext')
-    ft_corpus = annif.corpus.DocumentDirectory(ftdir, subject_index, 'fi')
+        os.path.dirname(__file__), "corpora", "archaeology", "fulltext"
+    )
+    ft_corpus = annif.corpus.DocumentDirectory(ftdir, subject_index, "fi")
     return ft_corpus
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def pretrained_vectors():
-    return py.path.local(os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'fasttext.vec'))
+    return py.path.local(
+        os.path.join(
+            os.path.dirname(__file__), "corpora", "archaeology", "fasttext.vec"
+        )
+    )
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def project(subject_index, datadir, registry, vocabulary):
     proj = unittest.mock.Mock()
-    proj.analyzer = annif.analyzer.get_analyzer('snowball(finnish)')
-    proj.language = 'fi'
+    proj.analyzer = annif.analyzer.get_analyzer("snowball(finnish)")
+    proj.language = "fi"
     proj.vocab = vocabulary
     proj.subjects = subject_index
     proj.datadir = str(datadir)
@@ -141,15 +129,15 @@ def project(subject_index, datadir, registry, vocabulary):
     return proj
 
 
-@pytest.fixture(scope='module')
+@pytest.fixture(scope="module")
 def app_project(app):
     with app.app_context():
-        dir = py.path.local(app.config['DATADIR'])
-        shutil.rmtree(os.path.join(str(dir), 'projects'), ignore_errors=True)
-        return annif.registry.get_project('dummy-en')
+        dir = py.path.local(app.config["DATADIR"])
+        shutil.rmtree(os.path.join(str(dir), "projects"), ignore_errors=True)
+        return annif.registry.get_project("dummy-en")
 
 
-@pytest.fixture(scope='function')
+@pytest.fixture(scope="function")
 def empty_corpus(tmpdir, subject_index):
-    empty_file = tmpdir.ensure('empty.tsv')
+    empty_file = tmpdir.ensure("empty.tsv")
     return annif.corpus.DocumentFile(str(empty_file), subject_index)
diff --git a/tests/test_analyzer.py b/tests/test_analyzer.py
index 261835ff8..cd7c2351d 100644
--- a/tests/test_analyzer.py
+++ b/tests/test_analyzer.py
@@ -60,16 +60,15 @@ def test_english_tokenize_words_no_filter():
 
 
 def test_english_filter_words_min_token():
-    analyzer = annif.analyzer.get_analyzer(
-        "snowball(english,token_min_length=2)")
+    analyzer = annif.analyzer.get_analyzer("snowball(english,token_min_length=2)")
     text = """Since 2000, a 3D printer can be used to print
     3 kinds of objects."""
     words = analyzer.tokenize_words(text)
     assert len(words) == 11
-    assert '2000' not in words
-    assert 'sinc' in words
-    assert 'object' in words
-    assert 'a' not in words
+    assert "2000" not in words
+    assert "sinc" in words
+    assert "object" in words
+    assert "a" not in words
 
 
 def test_swedish_analyzer_normalize_word():
@@ -91,6 +90,6 @@ def test_simple_analyzer():
 
 def test_simple_analyzer_token_size():
     analyzer = annif.analyzer.get_analyzer("simple(token_min_length=2)")
-    text = 'I do stuff'
+    text = "I do stuff"
     tokens = analyzer.tokenize_words(text)
     assert len(tokens) == 2
diff --git a/tests/test_analyzer_spacy.py b/tests/test_analyzer_spacy.py
index 7227fc38a..157dc7259 100644
--- a/tests/test_analyzer_spacy.py
+++ b/tests/test_analyzer_spacy.py
@@ -15,26 +15,65 @@ def test_spacy_model_not_found():
 
 def test_spacy_english_tokenize_words():
     analyzer = annif.analyzer.get_analyzer("spacy(en_core_web_sm)")
-    words = analyzer.tokenize_words("""
+    words = analyzer.tokenize_words(
+        """
         The quick brown foxes jumped over the lazy dogs in Paris.
-        """)
-    assert words == ['the', 'quick', 'brown', 'fox',
-                     'jump', 'over', 'the', 'lazy', 'dog', 'Paris']
+        """
+    )
+    assert words == [
+        "the",
+        "quick",
+        "brown",
+        "fox",
+        "jump",
+        "over",
+        "the",
+        "lazy",
+        "dog",
+        "Paris",
+    ]
 
 
 def test_spacy_english_tokenize_words_no_filter():
     analyzer = annif.analyzer.get_analyzer("spacy(en_core_web_sm)")
-    words = analyzer.tokenize_words("""
+    words = analyzer.tokenize_words(
+        """
         The quick brown foxes jumped over the lazy dogs in Paris.
-        """, filter=False)
-    assert words == ['the', 'quick', 'brown', 'fox',
-                     'jump', 'over', 'the', 'lazy', 'dog', 'in', 'Paris', '.']
+        """,
+        filter=False,
+    )
+    assert words == [
+        "the",
+        "quick",
+        "brown",
+        "fox",
+        "jump",
+        "over",
+        "the",
+        "lazy",
+        "dog",
+        "in",
+        "Paris",
+        ".",
+    ]
 
 
 def test_spacy_english_tokenize_words_lowercase():
     analyzer = annif.analyzer.get_analyzer("spacy(en_core_web_sm,lowercase=1)")
-    words = analyzer.tokenize_words("""
+    words = analyzer.tokenize_words(
+        """
         The quick brown foxes jumped over the lazy dogs in Paris.
-        """)
-    assert words == ['the', 'quick', 'brown', 'fox',
-                     'jump', 'over', 'the', 'lazy', 'dog', 'paris']
+        """
+    )
+    assert words == [
+        "the",
+        "quick",
+        "brown",
+        "fox",
+        "jump",
+        "over",
+        "the",
+        "lazy",
+        "dog",
+        "paris",
+    ]
diff --git a/tests/test_analyzer_voikko.py b/tests/test_analyzer_voikko.py
index 9d46d48be..cbab9a01a 100644
--- a/tests/test_analyzer_voikko.py
+++ b/tests/test_analyzer_voikko.py
@@ -9,7 +9,7 @@
 def test_voikko_getstate():
     analyzer = annif.analyzer.get_analyzer("voikko(fi)")
     state = analyzer.__getstate__()
-    assert state == {'param': 'fi', 'voikko': None}
+    assert state == {"param": "fi", "voikko": None}
 
 
 def test_voikko_finnish_analyzer_normalize_word():
diff --git a/tests/test_backend.py b/tests/test_backend.py
index 19d913773..35cf9ff1a 100644
--- a/tests/test_backend.py
+++ b/tests/test_backend.py
@@ -14,75 +14,78 @@ def test_get_backend_nonexistent():
 
 def test_get_backend_dummy(project, dummy_subject_index):
     dummy_type = annif.backend.get_backend("dummy")
-    dummy = dummy_type(backend_id='dummy', config_params={},
-                       project=project)
-    result = dummy.suggest(text='this is some text')
+    dummy = dummy_type(backend_id="dummy", config_params={}, project=project)
+    result = dummy.suggest(text="this is some text")
     assert len(result) == 1
     hits = result.as_list()
-    assert hits[0].subject_id == dummy_subject_index.by_uri(
-        'http://example.org/dummy')
+    assert hits[0].subject_id == dummy_subject_index.by_uri("http://example.org/dummy")
     assert hits[0].score == 1.0
 
 
 def test_learn_dummy(project, tmpdir):
     dummy_type = annif.backend.get_backend("dummy")
-    dummy = dummy_type(backend_id='dummy', config_params={},
-                       project=project)
+    dummy = dummy_type(backend_id="dummy", config_params={}, project=project)
 
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write(
-        '<http://www.yso.fi/onto/yso/p10849>\tarchaeologists')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write('<http://example.org/dummy>\tdummy')
-    docdir = annif.corpus.DocumentDirectory(
-        str(tmpdir), project.subjects, 'en')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("<http://www.yso.fi/onto/yso/p10849>\tarchaeologists")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("<http://example.org/dummy>\tdummy")
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), project.subjects, "en")
 
     dummy.learn(docdir)
 
-    result = dummy.suggest(text='this is some text')
+    result = dummy.suggest(text="this is some text")
     assert len(result) == 1
     hits = result.as_list()
     assert hits[0].subject_id is not None
     assert hits[0].subject_id == project.subjects.by_uri(
-        'http://www.yso.fi/onto/yso/p10849')
+        "http://www.yso.fi/onto/yso/p10849"
+    )
     assert hits[0].score == 1.0
 
 
 def test_fill_params_with_defaults(project):
-    dummy_type = annif.backend.get_backend('dummy')
-    dummy = dummy_type(backend_id='dummy', config_params={},
-                       project=project)
-    expected_default_params = {'limit': 100}
+    dummy_type = annif.backend.get_backend("dummy")
+    dummy = dummy_type(backend_id="dummy", config_params={}, project=project)
+    expected_default_params = {"limit": 100}
     assert expected_default_params == dummy.params
 
 
-@pytest.mark.skipif(importlib.util.find_spec("fasttext") is not None,
-                    reason="test requires that fastText is NOT installed")
+@pytest.mark.skipif(
+    importlib.util.find_spec("fasttext") is not None,
+    reason="test requires that fastText is NOT installed",
+)
 def test_get_backend_fasttext_not_installed():
     with pytest.raises(ValueError) as excinfo:
-        annif.backend.get_backend('fasttext')
-    assert 'fastText not available' in str(excinfo.value)
+        annif.backend.get_backend("fasttext")
+    assert "fastText not available" in str(excinfo.value)
 
 
-@pytest.mark.skipif(importlib.util.find_spec("tensorflow") is not None,
-                    reason="test requires that TensorFlow is NOT installed")
+@pytest.mark.skipif(
+    importlib.util.find_spec("tensorflow") is not None,
+    reason="test requires that TensorFlow is NOT installed",
+)
 def test_get_backend_nn_ensemble_not_installed():
     with pytest.raises(ValueError) as excinfo:
-        annif.backend.get_backend('nn_ensemble')
-    assert 'TensorFlow not available' in str(excinfo.value)
+        annif.backend.get_backend("nn_ensemble")
+    assert "TensorFlow not available" in str(excinfo.value)
 
 
-@pytest.mark.skipif(importlib.util.find_spec("omikuji") is not None,
-                    reason="test requires that Omikuji is NOT installed")
+@pytest.mark.skipif(
+    importlib.util.find_spec("omikuji") is not None,
+    reason="test requires that Omikuji is NOT installed",
+)
 def test_get_backend_omikuji_not_installed():
     with pytest.raises(ValueError) as excinfo:
-        annif.backend.get_backend('omikuji')
-    assert 'Omikuji not available' in str(excinfo.value)
+        annif.backend.get_backend("omikuji")
+    assert "Omikuji not available" in str(excinfo.value)
 
 
-@pytest.mark.skipif(importlib.util.find_spec("yake") is not None,
-                    reason="test requires that YAKE is NOT installed")
+@pytest.mark.skipif(
+    importlib.util.find_spec("yake") is not None,
+    reason="test requires that YAKE is NOT installed",
+)
 def test_get_backend_yake_not_installed():
     with pytest.raises(ValueError) as excinfo:
-        annif.backend.get_backend('yake')
-    assert 'YAKE not available' in str(excinfo.value)
+        annif.backend.get_backend("yake")
+    assert "YAKE not available" in str(excinfo.value)
diff --git a/tests/test_backend_ensemble.py b/tests/test_backend_ensemble.py
index 7355d1d03..fd7f25eb0 100644
--- a/tests/test_backend_ensemble.py
+++ b/tests/test_backend_ensemble.py
@@ -6,11 +6,10 @@
 
 
 def test_ensemble_train(project, document_corpus):
-    ensemble_type = annif.backend.get_backend('ensemble')
+    ensemble_type = annif.backend.get_backend("ensemble")
     ensemble = ensemble_type(
-        backend_id='ensemble',
-        config_params={'sources': 'dummy'},
-        project=project)
+        backend_id="ensemble", config_params={"sources": "dummy"}, project=project
+    )
 
     with pytest.raises(NotSupportedException):
         ensemble.train(document_corpus)
diff --git a/tests/test_backend_fasttext.py b/tests/test_backend_fasttext.py
index babbe5a29..f45beed44 100644
--- a/tests/test_backend_fasttext.py
+++ b/tests/test_backend_fasttext.py
@@ -11,18 +11,15 @@
 
 def test_fasttext_default_params(project):
     fasttext_type = annif.backend.get_backend("fasttext")
-    fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={},
-        project=project)
+    fasttext = fasttext_type(backend_id="fasttext", config_params={}, project=project)
 
     expected_default_params = {
-        'limit': 100,
-        'chunksize': 1,
-        'dim': 100,
-        'lr': 0.25,
-        'epoch': 5,
-        'loss': 'hs',
+        "limit": 100,
+        "chunksize": 1,
+        "dim": 100,
+        "lr": 0.25,
+        "epoch": 5,
+        "loss": "hs",
     }
     actual_params = fasttext.params
     for param, val in expected_default_params.items():
@@ -32,80 +29,65 @@ def test_fasttext_default_params(project):
 def test_fasttext_train(document_corpus, project, datadir):
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={
-            'limit': 50,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs'},
-        project=project)
+        backend_id="fasttext",
+        config_params={"limit": 50, "dim": 100, "lr": 0.25, "epoch": 20, "loss": "hs"},
+        project=project,
+    )
 
     fasttext.train(document_corpus)
     assert fasttext._model is not None
-    assert datadir.join('fasttext-model').exists()
-    assert datadir.join('fasttext-model').size() > 0
+    assert datadir.join("fasttext-model").exists()
+    assert datadir.join("fasttext-model").size() > 0
 
 
 def test_fasttext_train_cached_jobs(project, datadir):
-    assert datadir.join('fasttext-train.txt').exists()
-    datadir.join('fasttext-model').remove()
+    assert datadir.join("fasttext-train.txt").exists()
+    datadir.join("fasttext-model").remove()
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={
-            'limit': 50,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs'},
-        project=project)
+        backend_id="fasttext",
+        config_params={"limit": 50, "dim": 100, "lr": 0.25, "epoch": 20, "loss": "hs"},
+        project=project,
+    )
 
     fasttext.train("cached", jobs=2)
     assert fasttext._model is not None
-    assert datadir.join('fasttext-model').exists()
-    assert datadir.join('fasttext-model').size() > 0
+    assert datadir.join("fasttext-model").exists()
+    assert datadir.join("fasttext-model").size() > 0
 
 
 def test_fasttext_train_unknown_subject(tmpdir, datadir, project):
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={
-            'limit': 50,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs'},
-        project=project)
-
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("nonexistent\thttp://example.com/nonexistent\n" +
-                  "arkeologia\thttp://www.yso.fi/onto/yso/p1265")
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                project.subjects)
+        backend_id="fasttext",
+        config_params={"limit": 50, "dim": 100, "lr": 0.25, "epoch": 20, "loss": "hs"},
+        project=project,
+    )
+
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "nonexistent\thttp://example.com/nonexistent\n"
+        + "arkeologia\thttp://www.yso.fi/onto/yso/p1265"
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), project.subjects)
 
     fasttext.train(document_corpus)
     assert fasttext._model is not None
-    assert datadir.join('fasttext-model').exists()
-    assert datadir.join('fasttext-model').size() > 0
+    assert datadir.join("fasttext-model").exists()
+    assert datadir.join("fasttext-model").size() > 0
 
 
 def test_fasttext_train_nodocuments(project, empty_corpus):
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={
-            'limit': 50,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs'},
-        project=project)
+        backend_id="fasttext",
+        config_params={"limit": 50, "dim": 100, "lr": 0.25, "epoch": 20, "loss": "hs"},
+        project=project,
+    )
 
     with pytest.raises(NotSupportedException) as excinfo:
         fasttext.train(empty_corpus)
-    assert 'training backend fasttext with no documents' in str(excinfo.value)
+    assert "training backend fasttext with no documents" in str(excinfo.value)
 
 
 def test_train_fasttext_params(document_corpus, project, caplog):
@@ -114,19 +96,15 @@ def test_train_fasttext_params(document_corpus, project, caplog):
 
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
-        config_params={
-            'limit': 51,
-            'dim': 101,
-            'lr': 0.21,
-            'epoch': 21,
-            'loss': 'hs'},
-        project=project)
-    params = {'dim': 1, 'lr': 42.1, 'epoch': 0}
+        backend_id="fasttext",
+        config_params={"limit": 51, "dim": 101, "lr": 0.21, "epoch": 21, "loss": "hs"},
+        project=project,
+    )
+    params = {"dim": 1, "lr": 42.1, "epoch": 0}
 
     with caplog.at_level(logging.DEBUG):
         fasttext.train(document_corpus, params)
-    parameters_heading = 'Backend fasttext: Model parameters:'
+    parameters_heading = "Backend fasttext: Model parameters:"
     assert parameters_heading in caplog.text
     for line in caplog.text.splitlines():
         if parameters_heading in line:
@@ -135,45 +113,51 @@ def test_train_fasttext_params(document_corpus, project, caplog):
             assert "'epoch': 0" in line
 
 
-def test_fasttext_train_pretrained(datadir, document_corpus, project,
-                                   pretrained_vectors):
+def test_fasttext_train_pretrained(
+    datadir, document_corpus, project, pretrained_vectors
+):
     assert pretrained_vectors.exists()
     assert pretrained_vectors.size() > 0
 
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
+        backend_id="fasttext",
         config_params={
-            'limit': 50,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs',
-            'pretrainedVectors': str(pretrained_vectors)},
-        project=project)
+            "limit": 50,
+            "dim": 100,
+            "lr": 0.25,
+            "epoch": 20,
+            "loss": "hs",
+            "pretrainedVectors": str(pretrained_vectors),
+        },
+        project=project,
+    )
 
     fasttext.train(document_corpus)
     assert fasttext._model is not None
-    assert datadir.join('fasttext-model').exists()
-    assert datadir.join('fasttext-model').size() > 0
+    assert datadir.join("fasttext-model").exists()
+    assert datadir.join("fasttext-model").size() > 0
 
 
-def test_fasttext_train_pretrained_wrong_dim(datadir, document_corpus, project,
-                                             pretrained_vectors):
+def test_fasttext_train_pretrained_wrong_dim(
+    datadir, document_corpus, project, pretrained_vectors
+):
     assert pretrained_vectors.exists()
     assert pretrained_vectors.size() > 0
 
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
+        backend_id="fasttext",
         config_params={
-            'limit': 50,
-            'dim': 50,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs',
-            'pretrainedVectors': str(pretrained_vectors)},
-        project=project)
+            "limit": 50,
+            "dim": 50,
+            "lr": 0.25,
+            "epoch": 20,
+            "loss": "hs",
+            "pretrainedVectors": str(pretrained_vectors),
+        },
+        project=project,
+    )
 
     with pytest.raises(ValueError):
         fasttext.train(document_corpus)
@@ -183,24 +167,28 @@ def test_fasttext_train_pretrained_wrong_dim(datadir, document_corpus, project,
 def test_fasttext_suggest(project):
     fasttext_type = annif.backend.get_backend("fasttext")
     fasttext = fasttext_type(
-        backend_id='fasttext',
+        backend_id="fasttext",
         config_params={
-            'limit': 50,
-            'chunksize': 1,
-            'dim': 100,
-            'lr': 0.25,
-            'epoch': 20,
-            'loss': 'hs'},
-        project=project)
-
-    results = fasttext.suggest("""Arkeologiaa sanotaan joskus myös
+            "limit": 50,
+            "chunksize": 1,
+            "dim": 100,
+            "lr": 0.25,
+            "epoch": 20,
+            "loss": "hs",
+        },
+        project=project,
+    )
+
+    results = fasttext.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
     assert len(results) > 0
     hits = results.as_list()
-    archaeology = project.subjects.by_uri('http://www.yso.fi/onto/yso/p1265')
+    archaeology = project.subjects.by_uri("http://www.yso.fi/onto/yso/p1265")
     assert archaeology in [result.subject_id for result in hits]
diff --git a/tests/test_backend_http.py b/tests/test_backend_http.py
index dea6d44bb..c8cea6352 100644
--- a/tests/test_backend_http.py
+++ b/tests/test_backend_http.py
@@ -10,100 +10,119 @@
 
 
 def test_http_suggest(app_project):
-    with unittest.mock.patch('requests.post') as mock_request:
+    with unittest.mock.patch("requests.post") as mock_request:
         # create a mock response whose .json() method returns the list that we
         # define here
         mock_response = unittest.mock.Mock()
         mock_response.json.return_value = [
-            {'uri': 'http://example.org/dummy', 'label': 'dummy',
-             'score': 1.0}]
+            {"uri": "http://example.org/dummy", "label": "dummy", "score": 1.0}
+        ]
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=app_project)
-        result = http.suggest('this is some text')
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=app_project,
+        )
+        result = http.suggest("this is some text")
         assert len(result) == 1
         hits = result.as_list()
         assert hits[0].subject_id is not None
         assert hits[0].subject_id == app_project.subjects.by_uri(
-            'http://example.org/dummy')
+            "http://example.org/dummy"
+        )
         assert hits[0].score == 1.0
 
 
 def test_http_suggest_with_results(app_project):
-    with unittest.mock.patch('requests.post') as mock_request:
+    with unittest.mock.patch("requests.post") as mock_request:
         # create a mock response whose .json() method returns the list that we
         # define here
         mock_response = unittest.mock.Mock()
-        mock_response.json.return_value = {'results': [
-            {'uri': 'http://example.org/dummy-with-notation', 'label': 'dummy',
-             'notation': '42.42', 'score': 1.0}]}
+        mock_response.json.return_value = {
+            "results": [
+                {
+                    "uri": "http://example.org/dummy-with-notation",
+                    "label": "dummy",
+                    "notation": "42.42",
+                    "score": 1.0,
+                }
+            ]
+        }
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/dummy/analyze',
+                "endpoint": "http://api.example.org/dummy/analyze",
             },
-            project=app_project)
-        http.project.subjects.append(Subject(
-            uri='http://example.org/dummy-with-notation',
-            labels={'en': 'dummy', 'fi': 'dummy'},
-            notation='42.42'))
-
-        result = http.suggest('this is some text')
+            project=app_project,
+        )
+        http.project.subjects.append(
+            Subject(
+                uri="http://example.org/dummy-with-notation",
+                labels={"en": "dummy", "fi": "dummy"},
+                notation="42.42",
+            )
+        )
+
+        result = http.suggest("this is some text")
         assert len(result) == 1
         hits = result.as_list()
         assert hits[0].subject_id is not None
         assert hits[0].subject_id == http.project.subjects.by_uri(
-            'http://example.org/dummy-with-notation')
+            "http://example.org/dummy-with-notation"
+        )
         assert hits[0].score == 1.0
 
 
 def test_http_suggest_zero_score(project):
-    with unittest.mock.patch('requests.post') as mock_request:
+    with unittest.mock.patch("requests.post") as mock_request:
         # create a mock response whose .json() method returns the list that we
         # define here
         mock_response = unittest.mock.Mock()
         mock_response.json.return_value = [
-            {'uri': 'http://example.org/http', 'label': 'http', 'score': 0.0}]
+            {"uri": "http://example.org/http", "label": "http", "score": 0.0}
+        ]
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
-        result = http.suggest('this is some text')
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
+        result = http.suggest("this is some text")
         assert len(result) == 0
 
 
 def test_http_suggest_error(project):
-    with unittest.mock.patch('requests.post') as mock_request:
-        mock_request.side_effect = requests.exceptions.RequestException(
-            'failed')
+    with unittest.mock.patch("requests.post") as mock_request:
+        mock_request.side_effect = requests.exceptions.RequestException("failed")
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
-        result = http.suggest('this is some text')
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
+        result = http.suggest("this is some text")
         assert len(result) == 0
 
 
 def test_http_suggest_json_fails(project):
-    with unittest.mock.patch('requests.post') as mock_request:
+    with unittest.mock.patch("requests.post") as mock_request:
         # create a mock response whose .json() method returns the list that we
         # define here
         mock_response = unittest.mock.Mock()
@@ -112,17 +131,19 @@ def test_http_suggest_json_fails(project):
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
-        result = http.suggest('this is some text')
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
+        result = http.suggest("this is some text")
         assert len(result) == 0
 
 
 def test_http_suggest_unexpected_json(project):
-    with unittest.mock.patch('requests.post') as mock_request:
+    with unittest.mock.patch("requests.post") as mock_request:
         # create a mock response whose .json() method returns the list that we
         # define here
         mock_response = unittest.mock.Mock()
@@ -131,55 +152,63 @@ def test_http_suggest_unexpected_json(project):
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
-        result = http.suggest('this is some text')
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
+        result = http.suggest("this is some text")
         assert len(result) == 0
 
 
 def test_http_is_trained(project):
-    with unittest.mock.patch('requests.get') as mock_request:
+    with unittest.mock.patch("requests.get") as mock_request:
         # create a mock response whose .json() method returns the dict that we
         # define here
         mock_response = unittest.mock.Mock()
-        mock_response.json.return_value = {'is_trained': True}
+        mock_response.json.return_value = {"is_trained": True}
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
         assert http.is_trained
 
 
 def test_http_modification_time(project):
-    with unittest.mock.patch('requests.get') as mock_request:
+    with unittest.mock.patch("requests.get") as mock_request:
         # create a mock response whose .json() method returns the dict that we
         # define here
         mock_response = unittest.mock.Mock()
-        mock_response.json.return_value = {'modification_time':
-                                           '1970-01-01T00:00:00.000Z'}
+        mock_response.json.return_value = {
+            "modification_time": "1970-01-01T00:00:00.000Z"
+        }
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
         assert http.modification_time == datetime(
-            1970, 1, 1, 0, 0, 0, tzinfo=timezone.utc)
+            1970, 1, 1, 0, 0, 0, tzinfo=timezone.utc
+        )
 
 
 def test_http_modification_time_none(project):
-    with unittest.mock.patch('requests.get') as mock_request:
+    with unittest.mock.patch("requests.get") as mock_request:
         # create a mock response whose .json() method returns the dict that we
         # define here
         mock_response = unittest.mock.Mock()
@@ -188,42 +217,47 @@ def test_http_modification_time_none(project):
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
         assert http.modification_time is None
 
 
 def test_http_get_project_info_http_error(project):
-    with unittest.mock.patch('requests.get') as mock_request:
-        mock_request.side_effect = requests.exceptions.RequestException(
-            'failed')
+    with unittest.mock.patch("requests.get") as mock_request:
+        mock_request.side_effect = requests.exceptions.RequestException("failed")
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
         with pytest.raises(OperationFailedException):
-            http._get_project_info('is_trained')
+            http._get_project_info("is_trained")
 
 
 def test_http_get_project_info_json_decode_error(project):
-    with unittest.mock.patch('requests.get') as mock_request:
+    with unittest.mock.patch("requests.get") as mock_request:
         mock_response = unittest.mock.Mock()
         mock_response.json.side_effect = ValueError("JSON decode failed")
         mock_request.return_value = mock_response
 
         http_type = annif.backend.get_backend("http")
         http = http_type(
-            backend_id='http',
+            backend_id="http",
             config_params={
-                'endpoint': 'http://api.example.org/analyze',
-                'project': 'dummy'},
-            project=project)
+                "endpoint": "http://api.example.org/analyze",
+                "project": "dummy",
+            },
+            project=project,
+        )
         with pytest.raises(OperationFailedException):
-            http._get_project_info('is_trained')
+            http._get_project_info("is_trained")
diff --git a/tests/test_backend_mllm.py b/tests/test_backend_mllm.py
index 18e022c62..45c9e15b8 100644
--- a/tests/test_backend_mllm.py
+++ b/tests/test_backend_mllm.py
@@ -9,16 +9,13 @@
 
 def test_mllm_default_params(project):
     mllm_type = annif.backend.get_backend("mllm")
-    mllm = mllm_type(
-        backend_id='mllm',
-        config_params={},
-        project=project)
+    mllm = mllm_type(backend_id="mllm", config_params={}, project=project)
 
     expected_default_params = {
-        'limit': 100,  # from AnnifBackend class
-        'min_samples_leaf': 20,
-        'max_leaf_nodes': 1000,
-        'max_samples': 0.9
+        "limit": 100,  # from AnnifBackend class
+        "min_samples_leaf": 20,
+        "max_leaf_nodes": 1000,
+        "max_samples": 0.9,
     }
     actual_params = mllm.params
     for param, val in expected_default_params.items():
@@ -28,20 +25,21 @@ def test_mllm_default_params(project):
 def test_mllm_train(datadir, fulltext_corpus, project):
     mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 10, 'language': 'fi'},
-        project=project)
+        backend_id="mllm",
+        config_params={"limit": 10, "language": "fi"},
+        project=project,
+    )
 
     mllm.train(fulltext_corpus)
     assert mllm._model is not None
-    assert datadir.join('mllm-train.gz').exists()
-    assert datadir.join('mllm-train.gz').size() > 0
-    assert datadir.join('mllm-model.gz').exists()
-    assert datadir.join('mllm-model.gz').size() > 0
+    assert datadir.join("mllm-train.gz").exists()
+    assert datadir.join("mllm-train.gz").size() > 0
+    assert datadir.join("mllm-model.gz").exists()
+    assert datadir.join("mllm-model.gz").size() > 0
 
 
 def test_mllm_train_cached(datadir, project):
-    modelfile = datadir.join('mllm-model.gz')
+    modelfile = datadir.join("mllm-model.gz")
     assert modelfile.exists()
 
     old_size = modelfile.size()
@@ -49,9 +47,10 @@ def test_mllm_train_cached(datadir, project):
 
     mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 10, 'language': 'fi'},
-        project=project)
+        backend_id="mllm",
+        config_params={"limit": 10, "language": "fi"},
+        project=project,
+    )
 
     mllm.train("cached")
     assert mllm._model is not None
@@ -63,42 +62,43 @@ def test_mllm_train_cached(datadir, project):
 def test_mllm_train_nodocuments(project, empty_corpus):
     mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 10, 'language': 'fi'},
-        project=project)
+        backend_id="mllm",
+        config_params={"limit": 10, "language": "fi"},
+        project=project,
+    )
 
     with pytest.raises(NotSupportedException) as excinfo:
         mllm.train(empty_corpus)
-    assert 'training backend mllm with no documents' in str(excinfo.value)
+    assert "training backend mllm with no documents" in str(excinfo.value)
 
 
 def test_mllm_suggest(project):
-    mllm_type = annif.backend.get_backend('mllm')
+    mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="mllm", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
-    results = mllm.suggest("""Arkeologia on tieteenala, jota sanotaan joskus
+    results = mllm.suggest(
+        """Arkeologia on tieteenala, jota sanotaan joskus
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
     assert len(results) > 0
     assert len(results) <= 8
     hits = results.as_list()
-    archaeology = project.subjects.by_uri('http://www.yso.fi/onto/yso/p1265')
+    archaeology = project.subjects.by_uri("http://www.yso.fi/onto/yso/p1265")
     assert archaeology in [result.subject_id for result in hits]
 
 
 def test_mllm_suggest_no_matches(project):
-    mllm_type = annif.backend.get_backend('mllm')
+    mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="mllm", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
     results = mllm.suggest("Nothing matches this.")
 
@@ -106,39 +106,40 @@ def test_mllm_suggest_no_matches(project):
 
 
 def test_mllm_hyperopt(project, fulltext_corpus):
-    mllm_type = annif.backend.get_backend('mllm')
+    mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 10, 'language': 'fi'},
-        project=project)
+        backend_id="mllm",
+        config_params={"limit": 10, "language": "fi"},
+        project=project,
+    )
 
-    optimizer = mllm.get_hp_optimizer(fulltext_corpus, metric='NDCG')
+    optimizer = mllm.get_hp_optimizer(fulltext_corpus, metric="NDCG")
     optimizer.optimize(n_trials=3, n_jobs=1, results_file=None)
 
 
 def test_mllm_train_cached_no_data(datadir, project):
-    modelfile = datadir.join('mllm-model.gz')
+    modelfile = datadir.join("mllm-model.gz")
     assert modelfile.exists()
-    trainfile = datadir.join('mllm-train.gz')
+    trainfile = datadir.join("mllm-train.gz")
     trainfile.remove()
 
     mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 10, 'language': 'fi'},
-        project=project)
+        backend_id="mllm",
+        config_params={"limit": 10, "language": "fi"},
+        project=project,
+    )
 
     with pytest.raises(NotInitializedException):
         mllm.train("cached")
 
 
 def test_mllm_suggest_no_model(datadir, project):
-    mllm_type = annif.backend.get_backend('mllm')
+    mllm_type = annif.backend.get_backend("mllm")
     mllm = mllm_type(
-        backend_id='mllm',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="mllm", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
-    datadir.join('mllm-model.gz').remove()
+    datadir.join("mllm-model.gz").remove()
     with pytest.raises(NotInitializedException):
         mllm.suggest("example text")
diff --git a/tests/test_backend_nn_ensemble.py b/tests/test_backend_nn_ensemble.py
index 6bdf89a8a..7fd25c26e 100644
--- a/tests/test_backend_nn_ensemble.py
+++ b/tests/test_backend_nn_ensemble.py
@@ -13,16 +13,15 @@
 
 
 def test_lmdb_idx_to_key_to_idx():
-    assert annif.backend.nn_ensemble.idx_to_key(42) == b'00000042'
-    assert annif.backend.nn_ensemble.key_to_idx(b'00000042') == 42
+    assert annif.backend.nn_ensemble.idx_to_key(42) == b"00000042"
+    assert annif.backend.nn_ensemble.key_to_idx(b"00000042") == 42
 
 
 def test_nn_ensemble_initialize_parallel(project):
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=project)
+        backend_id="nn_ensemble", config_params={"sources": "dummy-en"}, project=project
+    )
 
     nn_ensemble.initialize(parallel=True)
     # model is still not loaded since we're preparing for parallel execution
@@ -30,68 +29,73 @@ def test_nn_ensemble_initialize_parallel(project):
 
 
 def test_nn_ensemble_suggest_no_model(project):
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=project)
+        backend_id="nn_ensemble", config_params={"sources": "dummy-en"}, project=project
+    )
 
     with pytest.raises(NotInitializedException):
         nn_ensemble.suggest("example text")
 
 
 def test_nn_ensemble_is_not_trained(app_project):
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
     assert not nn_ensemble.is_trained
 
 
 def test_nn_ensemble_can_set_lr(registry):
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'epochs': 1, 'lr': 0.002},
-        project=project)
-    nn_ensemble._create_model(['dummy-en'])
+        backend_id="nn_ensemble",
+        config_params={"epochs": 1, "lr": 0.002},
+        project=project,
+    )
+    nn_ensemble._create_model(["dummy-en"])
     assert nn_ensemble._model.optimizer.learning_rate.value() == 0.002
 
 
 def test_set_lmdb_map_size(registry, tmpdir):
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en', 'epochs': 1, 'lmdb_map_size': 1},
-        project=project)
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("dummy\thttp://example.org/dummy\n" +
-                  "another\thttp://example.org/dummy\n" +
-                  "none\thttp://example.org/none\n" * 40)
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                project.subjects)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en", "epochs": 1, "lmdb_map_size": 1},
+        project=project,
+    )
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "dummy\thttp://example.org/dummy\n"
+        + "another\thttp://example.org/dummy\n"
+        + "none\thttp://example.org/none\n" * 40
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), project.subjects)
 
     with pytest.raises(lmdb.MapFullError):
         nn_ensemble.train(document_corpus)
 
 
 def test_nn_ensemble_train_and_learn(registry, tmpdir):
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en', 'epochs': 1},
-        project=project)
-
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("dummy\thttp://example.org/dummy\n" +
-                  "another\thttp://example.org/dummy\n" +
-                  "none\thttp://example.org/none\n" * 40)
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                project.subjects)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en", "epochs": 1},
+        project=project,
+    )
+
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "dummy\thttp://example.org/dummy\n"
+        + "another\thttp://example.org/dummy\n"
+        + "none\thttp://example.org/none\n" * 40
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), project.subjects)
 
     nn_ensemble.train(document_corpus)
 
@@ -99,11 +103,11 @@ def test_nn_ensemble_train_and_learn(registry, tmpdir):
     assert nn_ensemble._model.optimizer.learning_rate.value() == 0.001
 
     datadir = py.path.local(project.datadir)
-    assert datadir.join('nn-model.h5').exists()
-    assert datadir.join('nn-model.h5').size() > 0
+    assert datadir.join("nn-model.h5").exists()
+    assert datadir.join("nn-model.h5").size() > 0
 
     # test online learning
-    modelfile = datadir.join('nn-model.h5')
+    modelfile = datadir.join("nn-model.h5")
 
     old_size = modelfile.size()
     old_mtime = modelfile.mtime()
@@ -111,8 +115,7 @@ def test_nn_ensemble_train_and_learn(registry, tmpdir):
     time.sleep(0.1)  # make sure the timestamp has a chance to increase
 
     # Learning is typically performed on one document at a time
-    document_corpus_single_doc = annif.corpus.LimitingDocumentCorpus(
-        document_corpus, 1)
+    document_corpus_single_doc = annif.corpus.LimitingDocumentCorpus(document_corpus, 1)
     nn_ensemble.learn(document_corpus_single_doc)
 
     assert modelfile.size() != old_size or modelfile.mtime() != old_mtime
@@ -120,75 +123,80 @@ def test_nn_ensemble_train_and_learn(registry, tmpdir):
 
 def test_nn_ensemble_train_cached(registry):
     # make sure we have the cached training data from the previous test
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     datadir = py.path.local(project.datadir)
-    assert datadir.join('nn-train.mdb').exists()
+    assert datadir.join("nn-train.mdb").exists()
 
-    datadir.join('nn-model.h5').remove()
+    datadir.join("nn-model.h5").remove()
 
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en', 'epochs': 2},
-        project=project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en", "epochs": 2},
+        project=project,
+    )
 
     nn_ensemble.train("cached")
 
-    assert datadir.join('nn-model.h5').exists()
-    assert datadir.join('nn-model.h5').size() > 0
+    assert datadir.join("nn-model.h5").exists()
+    assert datadir.join("nn-model.h5").size() > 0
 
 
 def test_nn_ensemble_train_and_learn_params(registry, tmpdir, capfd):
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en', 'epochs': 3},
-        project=project)
-
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("dummy\thttp://example.org/dummy\n" +
-                  "another\thttp://example.org/dummy\n" +
-                  "none\thttp://example.org/none")
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                project.subjects)
-
-    train_params = {'epochs': 3}
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en", "epochs": 3},
+        project=project,
+    )
+
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "dummy\thttp://example.org/dummy\n"
+        + "another\thttp://example.org/dummy\n"
+        + "none\thttp://example.org/none"
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), project.subjects)
+
+    train_params = {"epochs": 3}
     nn_ensemble.train(document_corpus, train_params)
     out, _ = capfd.readouterr()
-    assert 'Epoch 3/3' in out
+    assert "Epoch 3/3" in out
 
-    learn_params = {'learn-epochs': 2}
+    learn_params = {"learn-epochs": 2}
     nn_ensemble.learn(document_corpus, learn_params)
     out, _ = capfd.readouterr()
-    assert 'Epoch 2/2' in out
+    assert "Epoch 2/2" in out
 
 
 def test_nn_ensemble_is_trained(app_project):
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
     assert nn_ensemble.is_trained
 
 
 def test_nn_ensemble_modification_time(app_project):
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
-    assert datetime.now(timezone.utc) - \
-        nn_ensemble.modification_time < timedelta(1)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
+    assert datetime.now(timezone.utc) - nn_ensemble.modification_time < timedelta(1)
 
 
 def test_nn_ensemble_initialize(app_project):
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
 
     assert nn_ensemble._model is None
     nn_ensemble.initialize()
@@ -200,13 +208,14 @@ def test_nn_ensemble_initialize(app_project):
 def test_nn_ensemble_default_params(app_project):
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
 
     expected_default_params = {
-        'optimizer': 'adam',
-        'limit': 100,
+        "optimizer": "adam",
+        "limit": 100,
     }
     actual_params = nn_ensemble.params
     for param, val in expected_default_params.items():
@@ -214,11 +223,10 @@ def test_nn_ensemble_default_params(app_project):
 
 
 def test_nn_ensemble_train_nodocuments(project, empty_corpus):
-    nn_ensemble_type = annif.backend.get_backend('nn_ensemble')
+    nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=project)
+        backend_id="nn_ensemble", config_params={"sources": "dummy-en"}, project=project
+    )
     with pytest.raises(NotSupportedException):
         nn_ensemble.train(empty_corpus)
 
@@ -226,16 +234,19 @@ def test_nn_ensemble_train_nodocuments(project, empty_corpus):
 def test_nn_ensemble_suggest(app_project):
     nn_ensemble_type = annif.backend.get_backend("nn_ensemble")
     nn_ensemble = nn_ensemble_type(
-        backend_id='nn_ensemble',
-        config_params={'sources': 'dummy-en'},
-        project=app_project)
+        backend_id="nn_ensemble",
+        config_params={"sources": "dummy-en"},
+        project=app_project,
+    )
 
-    results = nn_ensemble.suggest("""Arkeologiaa sanotaan joskus myös
+    results = nn_ensemble.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen
         tiede tai oikeammin joukko tieteitä, jotka tutkivat ihmisen
         menneisyyttä. Tutkimusta tehdään analysoimalla muinaisjäännöksiä
         eli niitä jälkiä, joita ihmisten toiminta on jättänyt maaperään
-        tai vesistöjen pohjaan.""")
+        tai vesistöjen pohjaan."""
+    )
 
     assert nn_ensemble._model is not None
     assert len(results) > 0
diff --git a/tests/test_backend_omikuji.py b/tests/test_backend_omikuji.py
index 574ac205b..0888acef2 100644
--- a/tests/test_backend_omikuji.py
+++ b/tests/test_backend_omikuji.py
@@ -11,14 +11,11 @@
 
 def test_omikuji_default_params(project):
     omikuji_type = annif.backend.get_backend("omikuji")
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
 
     expected_default_params = {
-        'limit': 100,
-        'min_df': 1,
+        "limit": 100,
+        "min_df": 1,
     }
     actual_params = omikuji.params
     for param, val in expected_default_params.items():
@@ -26,32 +23,28 @@ def test_omikuji_default_params(project):
 
 
 def test_omikuji_suggest_no_vectorizer(project):
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
 
     with pytest.raises(NotInitializedException):
         omikuji.suggest("example text")
 
 
 def test_omikuji_create_train_file(tmpdir, project, datadir):
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("nonexistent\thttp://example.com/nonexistent\n" +
-                  "arkeologia\thttp://www.yso.fi/onto/yso/p1265\n" +
-                  "...\thttp://example.com/none")
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "nonexistent\thttp://example.com/nonexistent\n"
+        + "arkeologia\thttp://www.yso.fi/onto/yso/p1265\n"
+        + "...\thttp://example.com/none"
+    )
     corpus = annif.corpus.DocumentFile(str(tmpfile), project.subjects)
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
     input = (doc.text for doc in corpus.documents)
     veccorpus = omikuji.create_vectorizer(input, {})
     omikuji._create_train_file(veccorpus, corpus)
-    assert datadir.join('omikuji-train.txt').exists()
-    traindata = datadir.join('omikuji-train.txt').read().splitlines()
+    assert datadir.join("omikuji-train.txt").exists()
+    traindata = datadir.join("omikuji-train.txt").read().splitlines()
     assert len(traindata) == 2  # header + 1 example
     examples, features, labels = map(int, traindata[0].split())
     assert examples == 1
@@ -60,120 +53,104 @@ def test_omikuji_create_train_file(tmpdir, project, datadir):
 
 
 def test_omikuji_train(datadir, document_corpus, project):
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
 
     # verify that training works even if there is a preexisting model directory
     # - to simulate this we will create an empty directory instead
-    datadir.join('omikuji-model').ensure(dir=True)
-    assert not datadir.join('omikuji-model').listdir()  # empty dir
+    datadir.join("omikuji-model").ensure(dir=True)
+    assert not datadir.join("omikuji-model").listdir()  # empty dir
 
     omikuji.train(document_corpus)
     assert omikuji._model is not None
-    assert datadir.join('omikuji-model').exists()
-    assert datadir.join('omikuji-model').listdir()  # non-empty dir
+    assert datadir.join("omikuji-model").exists()
+    assert datadir.join("omikuji-model").listdir()  # non-empty dir
 
 
 def test_omikuji_train_ngram(datadir, document_corpus, project):
-    omikuji_type = annif.backend.get_backend('omikuji')
+    omikuji_type = annif.backend.get_backend("omikuji")
     omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={'ngram': 2},
-        project=project)
+        backend_id="omikuji", config_params={"ngram": 2}, project=project
+    )
 
-    datadir.join('omikuji-model').remove()
+    datadir.join("omikuji-model").remove()
     omikuji.train(document_corpus)
     assert omikuji._model is not None
-    assert datadir.join('omikuji-model').exists()
-    assert datadir.join('omikuji-model').listdir()  # non-empty dir
+    assert datadir.join("omikuji-model").exists()
+    assert datadir.join("omikuji-model").listdir()  # non-empty dir
 
 
 def test_omikuji_train_cached(datadir, project):
-    assert datadir.join('omikuji-train.txt').exists()
-    datadir.join('omikuji-model').remove()
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    assert datadir.join("omikuji-train.txt").exists()
+    datadir.join("omikuji-model").remove()
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
     omikuji.train("cached")
     assert omikuji._model is not None
-    assert datadir.join('omikuji-model').exists()
-    assert datadir.join('omikuji-model').listdir()  # non-empty dir
+    assert datadir.join("omikuji-model").exists()
+    assert datadir.join("omikuji-model").listdir()  # non-empty dir
 
 
 def test_omikuji_train_nodocuments(datadir, project, empty_corpus):
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
 
     with pytest.raises(NotSupportedException):
         omikuji.train(empty_corpus)
 
 
 def test_omikuji_train_params(datadir, document_corpus, project, capfd):
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
-    params = {'cluster_k': 1, 'max_depth': 2, 'collapse_every_n_layers': 42}
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
+    params = {"cluster_k": 1, "max_depth": 2, "collapse_every_n_layers": 42}
     omikuji.train(document_corpus, params)
 
     out, _ = capfd.readouterr()
-    parameters_heading = 'Training model with hyper-parameters HyperParam'
+    parameters_heading = "Training model with hyper-parameters HyperParam"
     assert parameters_heading in out
     for line in out.splitlines():
         if parameters_heading in line:
-            assert 'k: 1' in line
-            assert 'max_depth: 2' in line
-            assert 'collapse_every_n_layers: 42' in line
+            assert "k: 1" in line
+            assert "max_depth: 2" in line
+            assert "collapse_every_n_layers: 42" in line
 
 
 def test_omikuji_suggest(project):
-    omikuji_type = annif.backend.get_backend('omikuji')
+    omikuji_type = annif.backend.get_backend("omikuji")
     omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={'limit': 8},
-        project=project)
+        backend_id="omikuji", config_params={"limit": 8}, project=project
+    )
 
-    results = omikuji.suggest("""Arkeologiaa sanotaan joskus myös
+    results = omikuji.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
     assert len(results) > 0
     assert len(results) <= 8
     hits = results.as_list()
-    archaeology = project.subjects.by_uri('http://www.yso.fi/onto/yso/p1265')
+    archaeology = project.subjects.by_uri("http://www.yso.fi/onto/yso/p1265")
     assert archaeology in [result.subject_id for result in hits]
 
 
 def test_omikuji_suggest_no_input(project):
-    omikuji_type = annif.backend.get_backend('omikuji')
+    omikuji_type = annif.backend.get_backend("omikuji")
     omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={'limit': 8},
-        project=project)
+        backend_id="omikuji", config_params={"limit": 8}, project=project
+    )
 
     results = omikuji.suggest("j")
     assert len(results) == 0
 
 
 def test_omikuji_suggest_no_model(datadir, project):
-    omikuji_type = annif.backend.get_backend('omikuji')
-    omikuji = omikuji_type(
-        backend_id='omikuji',
-        config_params={},
-        project=project)
+    omikuji_type = annif.backend.get_backend("omikuji")
+    omikuji = omikuji_type(backend_id="omikuji", config_params={}, project=project)
 
-    datadir.join('omikuji-model').remove()
+    datadir.join("omikuji-model").remove()
     with pytest.raises(NotInitializedException):
         omikuji.suggest("example text")
diff --git a/tests/test_backend_pav.py b/tests/test_backend_pav.py
index 14ad8bbcd..38a32455b 100644
--- a/tests/test_backend_pav.py
+++ b/tests/test_backend_pav.py
@@ -11,14 +11,11 @@
 
 def test_pav_default_params(document_corpus, app_project):
     pav_type = annif.backend.get_backend("pav")
-    pav = pav_type(
-        backend_id='pav',
-        config_params={},
-        project=app_project)
+    pav = pav_type(backend_id="pav", config_params={}, project=app_project)
 
     expected_default_params = {
-        'min-docs': 10,
-        'limit': 100,
+        "min-docs": 10,
+        "limit": 100,
     }
     actual_params = pav.params
     for param, val in expected_default_params.items():
@@ -28,38 +25,42 @@ def test_pav_default_params(document_corpus, app_project):
 def test_pav_is_not_trained(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
     assert not pav.is_trained
 
 
 def test_pav_train(tmpdir, app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
-
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("dummy\thttp://example.org/dummy\n" +
-                  "another\thttp://example.org/dummy\n" +
-                  "none\thttp://example.org/none")
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                app_project.subjects)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
+
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "dummy\thttp://example.org/dummy\n"
+        + "another\thttp://example.org/dummy\n"
+        + "none\thttp://example.org/none"
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), app_project.subjects)
 
     pav.train(document_corpus)
     datadir = py.path.local(app_project.datadir)
-    assert datadir.join('pav-model-dummy-fi').exists()
-    assert datadir.join('pav-model-dummy-fi').size() > 0
+    assert datadir.join("pav-model-dummy-fi").exists()
+    assert datadir.join("pav-model-dummy-fi").size() > 0
 
 
 def test_pav_train_cached(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
 
     with pytest.raises(NotSupportedException):
         pav.train("cached")
@@ -68,21 +69,23 @@ def test_pav_train_cached(app_project):
 def test_pav_train_nodocuments(app_project, empty_corpus):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
 
     with pytest.raises(NotSupportedException) as excinfo:
         pav.train(empty_corpus)
-    assert 'training backend pav with no documents' in str(excinfo.value)
+    assert "training backend pav with no documents" in str(excinfo.value)
 
 
 def test_pav_initialize(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
 
     assert pav._models is None
     pav.initialize()
@@ -94,18 +97,21 @@ def test_pav_initialize(app_project):
 def test_pav_suggest(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
 
-    results = pav.suggest("""Arkeologiaa sanotaan joskus myös
+    results = pav.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
-    assert len(pav._models['dummy-fi']) == 1
+    assert len(pav._models["dummy-fi"]) == 1
     assert len(results) > 0
 
 
@@ -115,37 +121,41 @@ def test_pav_train_params(tmpdir, app_project, caplog):
 
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
-
-    tmpfile = tmpdir.join('document.tsv')
-    tmpfile.write("dummy\thttp://example.org/dummy\n" +
-                  "another\thttp://example.org/dummy\n" +
-                  "none\thttp://example.org/none")
-    document_corpus = annif.corpus.DocumentFile(str(tmpfile),
-                                                app_project.subjects)
-    params = {'min-docs': 5}
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
+
+    tmpfile = tmpdir.join("document.tsv")
+    tmpfile.write(
+        "dummy\thttp://example.org/dummy\n"
+        + "another\thttp://example.org/dummy\n"
+        + "none\thttp://example.org/none"
+    )
+    document_corpus = annif.corpus.DocumentFile(str(tmpfile), app_project.subjects)
+    params = {"min-docs": 5}
 
     with caplog.at_level(logging.DEBUG):
         pav.train(document_corpus, params)
-    parameters_spec = 'creating PAV model for source dummy-fi, min_docs=5'
+    parameters_spec = "creating PAV model for source dummy-fi, min_docs=5"
     assert parameters_spec in caplog.text
 
 
 def test_pav_is_trained(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
     assert pav.is_trained
 
 
 def test_pav_modification_time(app_project):
     pav_type = annif.backend.get_backend("pav")
     pav = pav_type(
-        backend_id='pav',
-        config_params={'limit': 50, 'min-docs': 2, 'sources': 'dummy-fi'},
-        project=app_project)
+        backend_id="pav",
+        config_params={"limit": 50, "min-docs": 2, "sources": "dummy-fi"},
+        project=app_project,
+    )
     assert datetime.now(timezone.utc) - pav.modification_time < timedelta(1)
diff --git a/tests/test_backend_stwfsa.py b/tests/test_backend_stwfsa.py
index 90043375b..174b11ae0 100644
--- a/tests/test_backend_stwfsa.py
+++ b/tests/test_backend_stwfsa.py
@@ -6,38 +6,32 @@
 
 
 _backend_conf = {
-    'language': 'fi',
-    'concept_type_uri': 'http://www.w3.org/2004/02/skos/core#Concept',
-    'sub_thesaurus_type_uri':
-        'http://www.w3.org/2004/02/skos/core#Collection',
-    'thesaurus_relation_type_uri':
-        'http://www.w3.org/2004/02/skos/core#member',
-    'thesaurus_relation_is_specialisation': True,
+    "language": "fi",
+    "concept_type_uri": "http://www.w3.org/2004/02/skos/core#Concept",
+    "sub_thesaurus_type_uri": "http://www.w3.org/2004/02/skos/core#Collection",
+    "thesaurus_relation_type_uri": "http://www.w3.org/2004/02/skos/core#member",
+    "thesaurus_relation_is_specialisation": True,
 }
 
 
 def test_stwfsa_default_params(project):
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params={},
-        project=project
+        backend_id=StwfsaBackend.name, config_params={}, project=project
     )
     expected_default_params = {
-        'concept_type_uri': 'http://www.w3.org/2004/02/skos/core#Concept',
-        'sub_thesaurus_type_uri':
-            'http://www.w3.org/2004/02/skos/core#Collection',
-        'thesaurus_relation_type_uri':
-            'http://www.w3.org/2004/02/skos/core#member',
-        'thesaurus_relation_is_specialisation': True,
-        'remove_deprecated': True,
-        'handle_title_case': True,
-        'extract_upper_case_from_braces': True,
-        'extract_any_case_from_braces': False,
-        'expand_ampersand_with_spaces': True,
-        'expand_abbreviation_with_punctuation': True,
-        'simple_english_plural_rules': False,
-        'use_txt_vec': False,
+        "concept_type_uri": "http://www.w3.org/2004/02/skos/core#Concept",
+        "sub_thesaurus_type_uri": "http://www.w3.org/2004/02/skos/core#Collection",
+        "thesaurus_relation_type_uri": "http://www.w3.org/2004/02/skos/core#member",
+        "thesaurus_relation_is_specialisation": True,
+        "remove_deprecated": True,
+        "handle_title_case": True,
+        "extract_upper_case_from_braces": True,
+        "extract_any_case_from_braces": False,
+        "expand_ampersand_with_spaces": True,
+        "expand_abbreviation_with_punctuation": True,
+        "simple_english_plural_rules": False,
+        "use_txt_vec": False,
     }
     actual_params = stwfsa.params
     assert expected_default_params == actual_params
@@ -45,10 +39,7 @@ def test_stwfsa_default_params(project):
 
 def test_stwfsa_not_initialized(project):
     stwfsa_type = get_backend(StwfsaBackend.name)
-    stwfsa = stwfsa_type(
-        backend_id='stwfsa',
-        config_params={},
-        project=project)
+    stwfsa = stwfsa_type(backend_id="stwfsa", config_params={}, project=project)
     with pytest.raises(NotInitializedException):
         stwfsa.suggest("example text")
 
@@ -56,9 +47,8 @@ def test_stwfsa_not_initialized(project):
 def test_stwfsa_train(document_corpus, project, datadir):
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params=_backend_conf,
-        project=project)
+        backend_id=StwfsaBackend.name, config_params=_backend_conf, project=project
+    )
     stwfsa.train(document_corpus)
     assert stwfsa._model is not None
     model_file = datadir.join(stwfsa.MODEL_FILE)
@@ -70,20 +60,18 @@ def test_empty_corpus(project):
     corpus = annif.corpus.DocumentList([])
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params=dict(),
-        project=project)
+        backend_id=StwfsaBackend.name, config_params=dict(), project=project
+    )
     with pytest.raises(NotSupportedException):
         stwfsa.train(corpus)
 
 
 def test_cached_corpus(project):
-    corpus = 'cached'
+    corpus = "cached"
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params=dict(),
-        project=project)
+        backend_id=StwfsaBackend.name, config_params=dict(), project=project
+    )
     with pytest.raises(NotSupportedException):
         stwfsa.train(corpus)
 
@@ -91,22 +79,21 @@ def test_cached_corpus(project):
 def test_stwfsa_suggest_unknown(project):
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params=dict(),
-        project=project)
-    results = stwfsa.suggest('1234')
+        backend_id=StwfsaBackend.name, config_params=dict(), project=project
+    )
+    results = stwfsa.suggest("1234")
     assert len(results) == 0
 
 
 def test_stwfsa_suggest(project, datadir):
     stwfsa_type = get_backend(StwfsaBackend.name)
     stwfsa = stwfsa_type(
-        backend_id=StwfsaBackend.name,
-        config_params=dict(),
-        project=project)
+        backend_id=StwfsaBackend.name, config_params=dict(), project=project
+    )
     # Just some randomly selected words, taken from YSO archaeology group.
     # And "random" words between them
-    results = stwfsa.suggest("""random
+    results = stwfsa.suggest(
+        """random
     muinais-DNA random random
     labyrintit random random random
     Eurooppalainen yleissopimus arkeologisen perinnön suojelusta random
@@ -116,8 +103,9 @@ def test_stwfsa_suggest(project, datadir):
     Mesa Verde random random random random
     muinaismuistoalueet  random random random
     zikkuratit random random
-    termoluminesenssi random random random""")
+    termoluminesenssi random random random"""
+    )
     assert len(results) == 10
     hits = results.as_list()
-    labyrinths = project.subjects.by_uri('http://www.yso.fi/onto/yso/p14174')
+    labyrinths = project.subjects.by_uri("http://www.yso.fi/onto/yso/p14174")
     assert labyrinths in [result.subject_id for result in hits]
diff --git a/tests/test_backend_svc.py b/tests/test_backend_svc.py
index e4bcd85c5..159310d7a 100644
--- a/tests/test_backend_svc.py
+++ b/tests/test_backend_svc.py
@@ -9,14 +9,11 @@
 
 def test_svc_default_params(project):
     svc_type = annif.backend.get_backend("svc")
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
     expected_default_params = {
-        'limit': 100,
-        'min_df': 1,
+        "limit": 100,
+        "min_df": 1,
     }
     actual_params = svc.params
     for param, val in expected_default_params.items():
@@ -24,99 +21,77 @@ def test_svc_default_params(project):
 
 
 def test_svc_suggest_no_vectorizer(project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
     with pytest.raises(NotInitializedException):
         svc.suggest("example text")
 
 
 def test_svc_train(datadir, document_corpus, project, caplog):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
     svc.train(document_corpus)
     assert svc._model is not None
-    assert datadir.join('svc-model.gz').exists()
-    assert 'training on a document with multiple subjects is not ' + \
-           'supported by SVC; selecting one random subject.' in caplog.text
+    assert datadir.join("svc-model.gz").exists()
+    assert (
+        "training on a document with multiple subjects is not "
+        + "supported by SVC; selecting one random subject."
+        in caplog.text
+    )
 
 
 def test_svc_train_ngram(datadir, document_corpus, project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={'ngram': 2},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={"ngram": 2}, project=project)
 
     svc.train(document_corpus)
     assert svc._model is not None
-    assert datadir.join('svc-model.gz').exists()
+    assert datadir.join("svc-model.gz").exists()
 
 
 def test_svc_train_cached(datadir, project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
     with pytest.raises(NotSupportedException):
         svc.train("cached")
 
 
 def test_svc_train_nodocuments(datadir, project, empty_corpus):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
     with pytest.raises(NotSupportedException):
         svc.train(empty_corpus)
 
 
 def test_svc_suggest(project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={'limit': 20},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={"limit": 20}, project=project)
 
     results = svc.suggest("""Arkeologiaa sanotaan joskus myös...""")
 
     assert len(results) > 0
     assert len(results) <= 20
     hits = results.as_list()
-    archaeologists = project.subjects.by_uri(
-        'http://www.yso.fi/onto/yso/p10849')
+    archaeologists = project.subjects.by_uri("http://www.yso.fi/onto/yso/p10849")
     assert archaeologists in [result.subject_id for result in hits]
 
 
 def test_svc_suggest_no_input(project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={'limit': 8},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={"limit": 8}, project=project)
 
     results = svc.suggest("j")
     assert len(results) == 0
 
 
 def test_svc_suggest_no_model(datadir, project):
-    svc_type = annif.backend.get_backend('svc')
-    svc = svc_type(
-        backend_id='svc',
-        config_params={},
-        project=project)
+    svc_type = annif.backend.get_backend("svc")
+    svc = svc_type(backend_id="svc", config_params={}, project=project)
 
-    datadir.join('svc-model.gz').remove()
+    datadir.join("svc-model.gz").remove()
     with pytest.raises(NotInitializedException):
         svc.suggest("example text")
diff --git a/tests/test_backend_tfidf.py b/tests/test_backend_tfidf.py
index 33b6ef5b9..23f180d6f 100644
--- a/tests/test_backend_tfidf.py
+++ b/tests/test_backend_tfidf.py
@@ -7,14 +7,9 @@
 
 def test_tfidf_default_params(project):
     tfidf_type = annif.backend.get_backend("tfidf")
-    tfidf = tfidf_type(
-        backend_id='tfidf',
-        config_params={},
-        project=project)
-
-    expected_default_params = {
-        'limit': 100  # From AnnifBackend class
-    }
+    tfidf = tfidf_type(backend_id="tfidf", config_params={}, project=project)
+
+    expected_default_params = {"limit": 100}  # From AnnifBackend class
     actual_params = tfidf.params
     for param, val in expected_default_params.items():
         assert param in actual_params and actual_params[param] == val
@@ -22,60 +17,53 @@ def test_tfidf_default_params(project):
 
 def test_tfidf_train(datadir, document_corpus, project):
     tfidf_type = annif.backend.get_backend("tfidf")
-    tfidf = tfidf_type(
-        backend_id='tfidf',
-        config_params={'limit': 10},
-        project=project)
+    tfidf = tfidf_type(backend_id="tfidf", config_params={"limit": 10}, project=project)
 
     tfidf.train(document_corpus)
     assert len(tfidf._index) > 0
-    assert datadir.join('tfidf-index').exists()
-    assert datadir.join('tfidf-index').size() > 0
+    assert datadir.join("tfidf-index").exists()
+    assert datadir.join("tfidf-index").size() > 0
 
 
 def test_tfidf_suggest(project):
     tfidf_type = annif.backend.get_backend("tfidf")
-    tfidf = tfidf_type(
-        backend_id='tfidf',
-        config_params={'limit': 10},
-        project=project)
+    tfidf = tfidf_type(backend_id="tfidf", config_params={"limit": 10}, project=project)
 
-    results = tfidf.suggest("""Arkeologiaa sanotaan joskus myös
+    results = tfidf.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
     assert len(results) == 10
     hits = results.as_list()
-    archaeology = project.subjects.by_uri('http://www.yso.fi/onto/yso/p1265')
+    archaeology = project.subjects.by_uri("http://www.yso.fi/onto/yso/p1265")
     assert archaeology in [result.subject_id for result in hits]
 
 
 def test_suggest_params(project):
     tfidf_type = annif.backend.get_backend("tfidf")
-    tfidf = tfidf_type(
-        backend_id='tfidf',
-        config_params={'limit': 10},
-        project=project)
-    params = {'limit': 3}
+    tfidf = tfidf_type(backend_id="tfidf", config_params={"limit": 10}, project=project)
+    params = {"limit": 3}
 
-    results = tfidf.suggest("""Arkeologiaa sanotaan joskus myös
+    results = tfidf.suggest(
+        """Arkeologiaa sanotaan joskus myös
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""", params)
+        pohjaan.""",
+        params,
+    )
     assert len(results) == 3
 
 
 def test_tfidf_suggest_unknown(project):
     tfidf_type = annif.backend.get_backend("tfidf")
-    tfidf = tfidf_type(
-        backend_id='tfidf',
-        config_params={'limit': 10},
-        project=project)
+    tfidf = tfidf_type(backend_id="tfidf", config_params={"limit": 10}, project=project)
 
     results = tfidf.suggest("abcdefghijk")  # unknown word
 
diff --git a/tests/test_backend_yake.py b/tests/test_backend_yake.py
index d05caacda..c0a5c9945 100644
--- a/tests/test_backend_yake.py
+++ b/tests/test_backend_yake.py
@@ -9,95 +9,98 @@
 
 
 def test_invalid_label_type(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'label_types': 'invalid_type', 'language': 'fi'},
-        project=project)
+        backend_id="yake",
+        config_params={"label_types": "invalid_type", "language": "fi"},
+        project=project,
+    )
     with pytest.raises(ConfigurationException):
         yake.suggest("example text")
 
 
 def test_yake_suggest(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
-    results = yake.suggest("""Arkeologia on tieteenala, jota sanotaan joskus
+    results = yake.suggest(
+        """Arkeologia on tieteenala, jota sanotaan joskus
         muinaistutkimukseksi tai muinaistieteeksi. Se on humanistinen tiede
         tai oikeammin joukko tieteitä, jotka tutkivat ihmisen menneisyyttä.
         Tutkimusta tehdään analysoimalla muinaisjäännöksiä eli niitä jälkiä,
         joita ihmisten toiminta on jättänyt maaperään tai vesistöjen
-        pohjaan.""")
+        pohjaan."""
+    )
 
     assert len(results) > 0
     assert len(results) <= 8
     hits = results.as_list()
-    archaeology = project.subjects.by_uri('http://www.yso.fi/onto/yso/p1265')
+    archaeology = project.subjects.by_uri("http://www.yso.fi/onto/yso/p1265")
     assert archaeology in [result.subject_id for result in hits]
 
 
 def test_yake_suggest_no_input(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
     results = yake.suggest("ja tai .,!")
     assert len(results) == 0
 
 
 def test_create_index_preflabels(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'language': 'fi', 'label_types': 'prefLabel'},
-        project=project)
+        backend_id="yake",
+        config_params={"language": "fi", "label_types": "prefLabel"},
+        project=project,
+    )
     index = yake._create_index()
     # Some of the 130 prefLabels get merged in lemmatization:
     # assyriologit, assyriologia (assyriolog); arkealogit, arkeologia
     # (arkeolog); egyptologit, egyptologia (egyptolog)
     assert len(index) == 127
-    assert 'kalliotaid' in index
-    assert 'luolamaalauks' not in index
+    assert "kalliotaid" in index
+    assert "luolamaalauks" not in index
 
 
 def test_create_index_pref_and_altlabels(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
     index = yake._create_index()
     assert len(index) == 160
-    assert 'kalliotaid' in index
-    assert 'luolamaalauks' in index
+    assert "kalliotaid" in index
+    assert "luolamaalauks" in index
 
 
 def test_create_index_altlabels(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'language': 'fi', 'label_types': 'altLabel'},
-        project=project)
+        backend_id="yake",
+        config_params={"language": "fi", "label_types": "altLabel"},
+        project=project,
+    )
     index = yake._create_index()
     assert len(index) == 33
-    assert 'kalliotaid' not in index
-    assert 'luolamaalauks' in index
+    assert "kalliotaid" not in index
+    assert "luolamaalauks" in index
 
 
 def test_combine_suggestions_different_uris(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
-    suggestions = [('http://www.yso.fi/onto/yso/p1265', 0.75),
-                   ('http://www.yso.fi/onto/yso/p1266', 0.25)]
+    suggestions = [
+        ("http://www.yso.fi/onto/yso/p1265", 0.75),
+        ("http://www.yso.fi/onto/yso/p1266", 0.25),
+    ]
     combined = yake._combine_suggestions(suggestions)
     assert len(combined) == 2
     assert combined[0] == suggestions[0]
@@ -105,24 +108,25 @@ def test_combine_suggestions_different_uris(project):
 
 
 def test_combine_suggestions_same_uri(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
     combined = yake._combine_suggestions(
-        [('http://www.yso.fi/onto/yso/p1265', 0.42),
-         ('http://www.yso.fi/onto/yso/p1265', 0.42)])
+        [
+            ("http://www.yso.fi/onto/yso/p1265", 0.42),
+            ("http://www.yso.fi/onto/yso/p1265", 0.42),
+        ]
+    )
     assert len(combined) == 1
 
 
 def test_combine_scores(project):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'limit': 8, 'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"limit": 8, "language": "fi"}, project=project
+    )
 
     assert yake._combine_scores(0.5, 0.5) == 0.8
     assert yake._combine_scores(0.75, 0.75) == 0.96
@@ -133,11 +137,10 @@ def test_combine_scores(project):
 
 
 def test_yake_train(project, document_corpus):
-    yake_type = annif.backend.get_backend('yake')
+    yake_type = annif.backend.get_backend("yake")
     yake = yake_type(
-        backend_id='yake',
-        config_params={'language': 'fi'},
-        project=project)
+        backend_id="yake", config_params={"language": "fi"}, project=project
+    )
 
     with pytest.raises(NotSupportedException):
         yake.train(document_corpus)
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 29a07aa47..06b50e7ee 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -10,12 +10,11 @@
 from click.testing import CliRunner
 import annif.cli
 
-runner = CliRunner(env={'ANNIF_CONFIG': 'annif.default_config.TestingConfig'})
+runner = CliRunner(env={"ANNIF_CONFIG": "annif.default_config.TestingConfig"})
 
 # Generate a random project name to use in tests
-TEMP_PROJECT = ''.join(
-    random.choice('abcdefghiklmnopqrstuvwxyz') for _ in range(8))
-PROJECTS_CONFIG_PATH = 'tests/projects_for_config_path_option.cfg'
+TEMP_PROJECT = "".join(random.choice("abcdefghiklmnopqrstuvwxyz") for _ in range(8))
+PROJECTS_CONFIG_PATH = "tests/projects_for_config_path_option.cfg"
 
 
 def test_list_projects():
@@ -23,394 +22,358 @@ def test_list_projects():
     assert not result.exception
     assert result.exit_code == 0
     # public project should be visible
-    assert 'dummy-fi' in result.output
+    assert "dummy-fi" in result.output
     # hidden project should be visible
-    assert 'dummy-en' in result.output
+    assert "dummy-en" in result.output
     # private project should be visible
-    assert 'dummy-private' in result.output
+    assert "dummy-private" in result.output
     # project with no access setting should be visible
-    assert 'ensemble' in result.output
+    assert "ensemble" in result.output
 
 
 def test_list_projects_bad_arguments():
     # The listprojects function does not accept any arguments, it should fail
     # if such are provided.
-    assert runner.invoke(
-        annif.cli.cli, [
-            'list-projects', 'moi']).exit_code != 0
-    assert runner.invoke(
-        annif.cli.run_list_projects, ['moi', '--debug', 'y']).exit_code != 0
+    assert runner.invoke(annif.cli.cli, ["list-projects", "moi"]).exit_code != 0
+    assert (
+        runner.invoke(annif.cli.run_list_projects, ["moi", "--debug", "y"]).exit_code
+        != 0
+    )
 
 
 def test_list_projects_config_path_option():
     result = runner.invoke(
-        annif.cli.cli, ["list-projects", "--projects", PROJECTS_CONFIG_PATH])
+        annif.cli.cli, ["list-projects", "--projects", PROJECTS_CONFIG_PATH]
+    )
     assert not result.exception
     assert result.exit_code == 0
-    assert 'dummy_for_projects_option' in result.output
-    assert 'dummy-fi' not in result.output
-    assert 'dummy-en' not in result.output
+    assert "dummy_for_projects_option" in result.output
+    assert "dummy-fi" not in result.output
+    assert "dummy-en" not in result.output
 
 
 def test_list_projects_config_path_option_nonexistent():
     failed_result = runner.invoke(
-        annif.cli.cli, ["list-projects", "--projects", "nonexistent.cfg"])
+        annif.cli.cli, ["list-projects", "--projects", "nonexistent.cfg"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Error: Invalid value for '-p' / '--projects': " \
-           "Path 'nonexistent.cfg' does not exist." in failed_result.output
+    assert (
+        "Error: Invalid value for '-p' / '--projects': "
+        "Path 'nonexistent.cfg' does not exist." in failed_result.output
+    )
 
 
 def test_show_project():
-    result = runner.invoke(annif.cli.cli, ['show-project', 'dummy-en'])
+    result = runner.invoke(annif.cli.cli, ["show-project", "dummy-en"])
     assert not result.exception
 
-    project_id = re.search(r'Project ID:\s+(.+)', result.output)
-    assert project_id.group(1) == 'dummy-en'
-    project_name = re.search(r'Project Name:\s+(.+)', result.output)
-    assert project_name.group(1) == 'Dummy English'
-    project_lang = re.search(r'Language:\s+(.+)', result.output)
-    assert project_lang.group(1) == 'en'
-    access = re.search(r'Access:\s+(.+)', result.output)
-    assert access.group(1) == 'hidden'
-    is_trained = re.search(r'Trained:\s+(.+)', result.output)
-    assert is_trained.group(1) == 'True'
-    modification_time = re.search(r'Modification time:\s+(.+)', result.output)
-    assert modification_time.group(1) == 'None'
+    project_id = re.search(r"Project ID:\s+(.+)", result.output)
+    assert project_id.group(1) == "dummy-en"
+    project_name = re.search(r"Project Name:\s+(.+)", result.output)
+    assert project_name.group(1) == "Dummy English"
+    project_lang = re.search(r"Language:\s+(.+)", result.output)
+    assert project_lang.group(1) == "en"
+    access = re.search(r"Access:\s+(.+)", result.output)
+    assert access.group(1) == "hidden"
+    is_trained = re.search(r"Trained:\s+(.+)", result.output)
+    assert is_trained.group(1) == "True"
+    modification_time = re.search(r"Modification time:\s+(.+)", result.output)
+    assert modification_time.group(1) == "None"
 
 
 def test_show_project_nonexistent():
-    assert runner.invoke(
-        annif.cli.cli,
-        ['show-project', TEMP_PROJECT]).exit_code != 0
+    assert runner.invoke(annif.cli.cli, ["show-project", TEMP_PROJECT]).exit_code != 0
     # Test should not fail even if the user queries for a non-existent project.
-    failed_result = runner.invoke(
-        annif.cli.cli, [
-            'show-project', 'nonexistent'])
+    failed_result = runner.invoke(annif.cli.cli, ["show-project", "nonexistent"])
     assert failed_result.exception
 
 
 def test_clear_project(testdatadir):
-    dirpath = os.path.join(str(testdatadir), 'projects', 'dummy-fi')
-    fpath = os.path.join(str(dirpath), 'test_clear_project_datafile')
+    dirpath = os.path.join(str(testdatadir), "projects", "dummy-fi")
+    fpath = os.path.join(str(dirpath), "test_clear_project_datafile")
     os.makedirs(dirpath)
-    open(fpath, 'a').close()
+    open(fpath, "a").close()
 
-    assert runner.invoke(
-        annif.cli.cli,
-        ['clear', 'dummy-fi']).exit_code == 0
+    assert runner.invoke(annif.cli.cli, ["clear", "dummy-fi"]).exit_code == 0
     assert not os.path.isdir(dirpath)
 
 
 def test_clear_project_nonexistent_data(testdatadir, caplog):
     logger = annif.logger
     logger.propagate = True
-    result = runner.invoke(annif.cli.cli, ['clear', 'dummy-fi'])
+    result = runner.invoke(annif.cli.cli, ["clear", "dummy-fi"])
     assert not result.exception
     assert result.exit_code == 0
     assert len(caplog.records) == 1
-    expected_msg = 'No model data to remove for project dummy-fi.'
+    expected_msg = "No model data to remove for project dummy-fi."
     assert expected_msg == caplog.records[0].message
 
 
 def test_list_vocabs_before_load(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        shutil.rmtree(str(testdatadir.join('vocabs/yso/')))
+        shutil.rmtree(str(testdatadir.join("vocabs/yso/")))
     result = runner.invoke(annif.cli.cli, ["list-vocabs"])
     assert not result.exception
     assert result.exit_code == 0
-    assert re.search(r'^yso\s+-\s+-\s+False',
-                     result.output, re.MULTILINE)
+    assert re.search(r"^yso\s+-\s+-\s+False", result.output, re.MULTILINE)
 
 
 def test_load_vocab_csv(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.csv')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.csv")))
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.ttl')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.ttl")))
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'subjects.csv')
-    result = runner.invoke(annif.cli.cli,
-                           ['load-vocab', 'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "subjects.csv"
+    )
+    result = runner.invoke(annif.cli.cli, ["load-vocab", "yso", subjectfile])
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.ttl').exists()
-    assert testdatadir.join('vocabs/yso/subjects.ttl').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').exists()
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.ttl").exists()
+    assert testdatadir.join("vocabs/yso/subjects.ttl").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").exists()
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").size() > 0
 
 
 def test_load_vocab_tsv(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.csv')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.csv")))
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.ttl')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.ttl")))
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'subjects.tsv')
-    result = runner.invoke(annif.cli.cli,
-                           ['load-vocab', '--language', 'fi',
-                            'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "subjects.tsv"
+    )
+    result = runner.invoke(
+        annif.cli.cli, ["load-vocab", "--language", "fi", "yso", subjectfile]
+    )
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.ttl').exists()
-    assert testdatadir.join('vocabs/yso/subjects.ttl').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').exists()
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.ttl").exists()
+    assert testdatadir.join("vocabs/yso/subjects.ttl").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").exists()
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").size() > 0
 
 
 def test_load_vocab_tsv_no_lang(testdatadir):
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'subjects.tsv')
-    failed_result = runner.invoke(annif.cli.cli,
-                                  ['load-vocab', 'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "subjects.tsv"
+    )
+    failed_result = runner.invoke(annif.cli.cli, ["load-vocab", "yso", subjectfile])
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Please use --language option to set the language " \
+    assert (
+        "Please use --language option to set the language "
         "of a TSV vocabulary." in failed_result.output
+    )
 
 
 def test_load_vocab_tsv_with_bom(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.csv')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.csv")))
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.ttl')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.ttl")))
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'subjects-bom.tsv')
-    result = runner.invoke(annif.cli.cli,
-                           ['load-vocab', '--language', 'fi',
-                            'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "subjects-bom.tsv"
+    )
+    result = runner.invoke(
+        annif.cli.cli, ["load-vocab", "--language", "fi", "yso", subjectfile]
+    )
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.ttl').exists()
-    assert testdatadir.join('vocabs/yso/subjects.ttl').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').exists()
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.ttl").exists()
+    assert testdatadir.join("vocabs/yso/subjects.ttl").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").exists()
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").size() > 0
 
 
 def test_load_vocab_rdf(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.csv')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.csv")))
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.ttl')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.ttl")))
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'yso-archaeology.rdf')
-    result = runner.invoke(annif.cli.cli,
-                           ['load-vocab', 'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.rdf"
+    )
+    result = runner.invoke(annif.cli.cli, ["load-vocab", "yso", subjectfile])
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.ttl').exists()
-    assert testdatadir.join('vocabs/yso/subjects.ttl').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').exists()
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.ttl").exists()
+    assert testdatadir.join("vocabs/yso/subjects.ttl").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").exists()
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").size() > 0
 
 
 def test_load_vocab_ttl(testdatadir):
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.csv')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.csv")))
     with contextlib.suppress(FileNotFoundError):
-        os.remove(str(testdatadir.join('vocabs/yso/subjects.ttl')))
+        os.remove(str(testdatadir.join("vocabs/yso/subjects.ttl")))
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'yso-archaeology.ttl')
-    result = runner.invoke(annif.cli.cli,
-                           ['load-vocab', 'yso', subjectfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
+    )
+    result = runner.invoke(annif.cli.cli, ["load-vocab", "yso", subjectfile])
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.ttl').exists()
-    assert testdatadir.join('vocabs/yso/subjects.ttl').size() > 0
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').exists()
-    assert testdatadir.join('vocabs/yso/subjects.dump.gz').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.ttl").exists()
+    assert testdatadir.join("vocabs/yso/subjects.ttl").size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").exists()
+    assert testdatadir.join("vocabs/yso/subjects.dump.gz").size() > 0
 
 
 def test_load_vocab_nonexistent_vocab():
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'yso-archaeology.ttl')
+        os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
+    )
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'load-vocab', 'notfound', subjectfile])
+        annif.cli.cli, ["load-vocab", "notfound", subjectfile]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "No vocabularies found with the id 'notfound'." \
-        in failed_result.output
+    assert "No vocabularies found with the id 'notfound'." in failed_result.output
 
 
 def test_load_vocab_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'load-vocab', 'dummy', 'nonexistent_path'])
+        annif.cli.cli, ["load-vocab", "dummy", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for 'SUBJECTFILE': " \
-           "File 'nonexistent_path' does not exist." in failed_result.output
+    assert (
+        "Invalid value for 'SUBJECTFILE': "
+        "File 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_list_vocabs_after_load():
     result = runner.invoke(annif.cli.cli, ["list-vocabs"])
     assert not result.exception
     assert result.exit_code == 0
-    assert re.search(r'^dummy\s+en,fi\s+2\s+True',
-                     result.output, re.MULTILINE)
-    assert re.search(r'^yso\s+en,fi,sv\s+130\s+True',
-                     result.output, re.MULTILINE)
+    assert re.search(r"^dummy\s+en,fi\s+2\s+True", result.output, re.MULTILINE)
+    assert re.search(r"^yso\s+en,fi,sv\s+130\s+True", result.output, re.MULTILINE)
 
 
 def test_train(testdatadir):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli, ['train', 'tfidf-fi', docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["train", "tfidf-fi", docfile])
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('projects/tfidf-fi/vectorizer').exists()
-    assert testdatadir.join('projects/tfidf-fi/vectorizer').size() > 0
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').exists()
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').size() > 0
+    assert testdatadir.join("projects/tfidf-fi/vectorizer").exists()
+    assert testdatadir.join("projects/tfidf-fi/vectorizer").size() > 0
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").exists()
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").size() > 0
 
 
 def test_train_multiple(testdatadir):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli,
-                           ['train', 'tfidf-fi', docfile, docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["train", "tfidf-fi", docfile, docfile])
     assert not result.exception
     assert result.exit_code == 0
-    assert testdatadir.join('projects/tfidf-fi/vectorizer').exists()
-    assert testdatadir.join('projects/tfidf-fi/vectorizer').size() > 0
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').exists()
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').size() > 0
+    assert testdatadir.join("projects/tfidf-fi/vectorizer").exists()
+    assert testdatadir.join("projects/tfidf-fi/vectorizer").size() > 0
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").exists()
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").size() > 0
 
 
 def test_train_cached(testdatadir):
-    result = runner.invoke(annif.cli.cli,
-                           ['train', '--cached', 'tfidf-fi'])
+    result = runner.invoke(annif.cli.cli, ["train", "--cached", "tfidf-fi"])
     assert result.exception
     assert result.exit_code == 1
-    assert 'Training tfidf project from cached data not supported.' \
-           in result.output
+    assert "Training tfidf project from cached data not supported." in result.output
 
 
 def test_train_cached_with_corpus(testdatadir):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli,
-                           ['train', '--cached', 'tfidf-fi', docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["train", "--cached", "tfidf-fi", docfile])
     assert result.exception
     assert result.exit_code == 2
-    assert 'Corpus paths cannot be given when using --cached option.' \
-           in result.output
+    assert "Corpus paths cannot be given when using --cached option." in result.output
 
 
 def test_train_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'train', 'dummy-fi', 'nonexistent_path'])
+        annif.cli.cli, ["train", "dummy-fi", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for '[PATHS]...': " \
-           "Path 'nonexistent_path' does not exist." in failed_result.output
+    assert (
+        "Invalid value for '[PATHS]...': "
+        "Path 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_train_no_path(caplog):
     logger = annif.logger
     logger.propagate = True
-    result = runner.invoke(
-        annif.cli.cli, [
-            'train', 'dummy-fi'])
+    result = runner.invoke(annif.cli.cli, ["train", "dummy-fi"])
     assert not result.exception
     assert result.exit_code == 0
-    assert 'Reading empty file' == caplog.records[0].message
+    assert "Reading empty file" == caplog.records[0].message
 
 
 def test_train_docslimit_zero():
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'train', 'tfidf-fi', docfile, '--docs-limit', '0'])
+        annif.cli.cli, ["train", "tfidf-fi", docfile, "--docs-limit", "0"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Not supported: Cannot train tfidf project with no documents"  \
+    assert (
+        "Not supported: Cannot train tfidf project with no documents"
         in failed_result.output
+    )
 
 
 def test_learn(testdatadir):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli, ['learn', 'dummy-fi', docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["learn", "dummy-fi", docfile])
     assert not result.exception
     assert result.exit_code == 0
 
 
 def test_learn_notsupported(testdatadir):
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli, ['learn', 'tfidf-fi', docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["learn", "tfidf-fi", docfile])
     assert result.exit_code != 0
-    assert 'Learning not supported' in result.output
+    assert "Learning not supported" in result.output
 
 
 def test_learn_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'learn', 'dummy-fi', 'nonexistent_path'])
+        annif.cli.cli, ["learn", "dummy-fi", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for '[PATHS]...': " \
-           "Path 'nonexistent_path' does not exist." in failed_result.output
+    assert (
+        "Invalid value for '[PATHS]...': "
+        "Path 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_suggest():
-    result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', 'dummy-fi'],
-        input='kissa')
+    result = runner.invoke(annif.cli.cli, ["suggest", "dummy-fi"], input="kissa")
     assert not result.exception
     assert result.output == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
     assert result.exit_code == 0
@@ -418,9 +381,8 @@ def test_suggest():
 
 def test_suggest_with_language_override():
     result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', '--language', 'en', 'dummy-fi'],
-        input='kissa')
+        annif.cli.cli, ["suggest", "--language", "en", "dummy-fi"], input="kissa"
+    )
     assert not result.exception
     assert result.output == "<http://example.org/dummy>\tdummy\t1.0\n"
     assert result.exit_code == 0
@@ -428,9 +390,8 @@ def test_suggest_with_language_override():
 
 def test_suggest_with_language_override_bad_value():
     failed_result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', '--language', 'xx', 'dummy-fi'],
-        input='kissa')
+        annif.cli.cli, ["suggest", "--language", "xx", "dummy-fi"], input="kissa"
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
     assert 'language "xx" not supported by vocabulary' in failed_result.output
@@ -440,9 +401,8 @@ def test_suggest_with_different_vocab_language():
     # project language is English - input should be in English
     # vocab language is Finnish - subject labels should be in Finnish
     result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', 'dummy-vocablang'],
-        input='the cat sat on the mat')
+        annif.cli.cli, ["suggest", "dummy-vocablang"], input="the cat sat on the mat"
+    )
     assert not result.exception
     assert result.output == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
     assert result.exit_code == 0
@@ -451,30 +411,27 @@ def test_suggest_with_different_vocab_language():
 def test_suggest_with_notations():
     result = runner.invoke(
         annif.cli.cli,
-        ['suggest',
-         '--backend-param', 'dummy.uri=http://example.org/none', 'dummy-fi'],
-        input='kissa')
+        ["suggest", "--backend-param", "dummy.uri=http://example.org/none", "dummy-fi"],
+        input="kissa",
+    )
     assert not result.exception
     assert result.output == "<http://example.org/none>\tnone-fi\t42.42\t1.0\n"
     assert result.exit_code == 0
 
 
 def test_suggest_nonexistent():
-    result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', TEMP_PROJECT],
-        input='kissa')
+    result = runner.invoke(annif.cli.cli, ["suggest", TEMP_PROJECT], input="kissa")
     assert result.exception
-    assert result.output == "No projects found with id '{}'.\n".format(
-        TEMP_PROJECT)
+    assert result.output == "No projects found with id '{}'.\n".format(TEMP_PROJECT)
     assert result.exit_code != 0
 
 
 def test_suggest_param():
     result = runner.invoke(
         annif.cli.cli,
-        ['suggest', '--backend-param', 'dummy.score=0.8', 'dummy-fi'],
-        input='kissa')
+        ["suggest", "--backend-param", "dummy.score=0.8", "dummy-fi"],
+        input="kissa",
+    )
     assert not result.exception
     assert result.output == "<http://example.org/dummy>\tdummy-fi\t0.8\n"
     assert result.exit_code == 0
@@ -483,71 +440,79 @@ def test_suggest_param():
 def test_suggest_param_backend_nonexistent():
     result = runner.invoke(
         annif.cli.cli,
-        ['suggest', '--backend-param', 'not_a_backend.score=0.8', 'dummy-fi'],
-        input='kissa')
+        ["suggest", "--backend-param", "not_a_backend.score=0.8", "dummy-fi"],
+        input="kissa",
+    )
     assert result.exception
-    assert 'The backend not_a_backend in CLI option ' + \
-        '"-b not_a_backend.score=0.8" not matching the project backend ' + \
-        'dummy.' in result.output
+    assert (
+        "The backend not_a_backend in CLI option "
+        + '"-b not_a_backend.score=0.8" not matching the project backend '
+        + "dummy."
+        in result.output
+    )
     assert result.exit_code != 0
 
 
 def test_suggest_ensemble():
     result = runner.invoke(
-        annif.cli.cli,
-        ['suggest', 'ensemble'],
-        input='the cat sat on the mat')
+        annif.cli.cli, ["suggest", "ensemble"], input="the cat sat on the mat"
+    )
     assert not result.exception
     assert result.output == "<http://example.org/dummy>\tdummy\t1.0\n"
     assert result.exit_code == 0
 
 
 def test_index(tmpdir):
-    tmpdir.join('doc1.txt').write('nothing special')
+    tmpdir.join("doc1.txt").write("nothing special")
 
-    result = runner.invoke(
-        annif.cli.cli, ['index', 'dummy-en', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["index", "dummy-en", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
 
-    assert tmpdir.join('doc1.annif').exists()
-    assert tmpdir.join('doc1.annif').read_text(
-        'utf-8') == "<http://example.org/dummy>\tdummy\t1.0\n"
+    assert tmpdir.join("doc1.annif").exists()
+    assert (
+        tmpdir.join("doc1.annif").read_text("utf-8")
+        == "<http://example.org/dummy>\tdummy\t1.0\n"
+    )
 
     # make sure that preexisting subject files are not overwritten
-    result = runner.invoke(
-        annif.cli.cli, ['index', 'dummy-en', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["index", "dummy-en", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
     assert "Not overwriting" in result.output
 
     # check that the --force parameter forces overwriting
-    result = runner.invoke(
-        annif.cli.cli, ['index', 'dummy-fi', '--force', str(tmpdir)])
-    assert tmpdir.join('doc1.annif').exists()
+    result = runner.invoke(annif.cli.cli, ["index", "dummy-fi", "--force", str(tmpdir)])
+    assert tmpdir.join("doc1.annif").exists()
     assert "Not overwriting" not in result.output
-    assert tmpdir.join('doc1.annif').read_text(
-        'utf-8') == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
+    assert (
+        tmpdir.join("doc1.annif").read_text("utf-8")
+        == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
+    )
 
 
 def test_index_with_language_override(tmpdir):
-    tmpdir.join('doc1.txt').write('nothing special')
+    tmpdir.join("doc1.txt").write("nothing special")
 
     result = runner.invoke(
-        annif.cli.cli, ['index', '--language', 'fi', 'dummy-en', str(tmpdir)])
+        annif.cli.cli, ["index", "--language", "fi", "dummy-en", str(tmpdir)]
+    )
     assert not result.exception
     assert result.exit_code == 0
 
-    assert tmpdir.join('doc1.annif').exists()
-    assert tmpdir.join('doc1.annif').read_text(
-        'utf-8') == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
+    assert tmpdir.join("doc1.annif").exists()
+    assert (
+        tmpdir.join("doc1.annif").read_text("utf-8")
+        == "<http://example.org/dummy>\tdummy-fi\t1.0\n"
+    )
 
 
 def test_index_with_language_override_bad_value(tmpdir):
-    tmpdir.join('doc1.txt').write('nothing special')
+    tmpdir.join("doc1.txt").write("nothing special")
 
     failed_result = runner.invoke(
-        annif.cli.cli, ['index', '--language', 'xx', 'dummy-en', str(tmpdir)])
+        annif.cli.cli, ["index", "--language", "xx", "dummy-en", str(tmpdir)]
+    )
 
     assert failed_result.exception
     assert failed_result.exit_code != 0
@@ -556,194 +521,200 @@ def test_index_with_language_override_bad_value(tmpdir):
 
 def test_index_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'index', 'dummy-fi', 'nonexistent_path'])
+        annif.cli.cli, ["index", "dummy-fi", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for 'DIRECTORY': " \
-           "Directory 'nonexistent_path' does not exist." \
-           in failed_result.output
+    assert (
+        "Invalid value for 'DIRECTORY': "
+        "Directory 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_eval_label(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
 
-    result = runner.invoke(annif.cli.cli, ['eval', 'dummy-en', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["eval", "dummy-en", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
 
-    precision = re.search(r'Precision .*doc.*:\s+(\d.\d+)', result.output)
+    precision = re.search(r"Precision .*doc.*:\s+(\d.\d+)", result.output)
     assert float(precision.group(1)) == 0.5
-    recall = re.search(r'Recall .*doc.*:\s+(\d.\d+)', result.output)
+    recall = re.search(r"Recall .*doc.*:\s+(\d.\d+)", result.output)
     assert float(recall.group(1)) == 0.5
-    f_measure = re.search(r'F1 score .*doc.*:\s+(\d.\d+)', result.output)
+    f_measure = re.search(r"F1 score .*doc.*:\s+(\d.\d+)", result.output)
     assert float(f_measure.group(1)) == 0.5
-    precision1 = re.search(r'Precision@1:\s+(\d.\d+)', result.output)
+    precision1 = re.search(r"Precision@1:\s+(\d.\d+)", result.output)
     assert float(precision1.group(1)) == 0.5
-    precision3 = re.search(r'Precision@3:\s+(\d.\d+)', result.output)
+    precision3 = re.search(r"Precision@3:\s+(\d.\d+)", result.output)
     assert float(precision3.group(1)) == 0.5
-    precision5 = re.search(r'Precision@5:\s+(\d.\d+)', result.output)
+    precision5 = re.search(r"Precision@5:\s+(\d.\d+)", result.output)
     assert float(precision5.group(1)) == 0.5
-    lrap = re.search(r'LRAP:\s+(\d.\d+)', result.output)
+    lrap = re.search(r"LRAP:\s+(\d.\d+)", result.output)
     assert float(lrap.group(1)) == 0.75
-    true_positives = re.search(r'True positives:\s+(\d+)', result.output)
+    true_positives = re.search(r"True positives:\s+(\d+)", result.output)
     assert int(true_positives.group(1)) == 1
-    false_positives = re.search(r'False positives:\s+(\d+)', result.output)
+    false_positives = re.search(r"False positives:\s+(\d+)", result.output)
     assert int(false_positives.group(1)) == 1
-    false_negatives = re.search(r'False negatives:\s+(\d+)', result.output)
+    false_negatives = re.search(r"False negatives:\s+(\d+)", result.output)
     assert int(false_negatives.group(1)) == 1
-    ndocs = re.search(r'Documents evaluated:\s+(\d+)', result.output)
+    ndocs = re.search(r"Documents evaluated:\s+(\d+)", result.output)
     assert int(ndocs.group(1)) == 2
 
 
 def test_eval_uri(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write("<http://example.org/dummy>\tdummy\n")
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write("<http://example.org/none>\tnone\n")
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("<http://example.org/dummy>\tdummy\n")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("<http://example.org/none>\tnone\n")
+    tmpdir.join("doc3.txt").write("doc3")
 
-    result = runner.invoke(annif.cli.cli, ['eval', 'dummy-en', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["eval", "dummy-en", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
 
-    precision = re.search(r'Precision .*doc.*:\s+(\d.\d+)', result.output)
+    precision = re.search(r"Precision .*doc.*:\s+(\d.\d+)", result.output)
     assert float(precision.group(1)) == 0.5
-    recall = re.search(r'Recall .*doc.*:\s+(\d.\d+)', result.output)
+    recall = re.search(r"Recall .*doc.*:\s+(\d.\d+)", result.output)
     assert float(recall.group(1)) == 0.5
-    f_measure = re.search(r'F1 score .*doc.*:\s+(\d.\d+)', result.output)
+    f_measure = re.search(r"F1 score .*doc.*:\s+(\d.\d+)", result.output)
     assert float(f_measure.group(1)) == 0.5
-    precision1 = re.search(r'Precision@1:\s+(\d.\d+)', result.output)
+    precision1 = re.search(r"Precision@1:\s+(\d.\d+)", result.output)
     assert float(precision1.group(1)) == 0.5
-    precision3 = re.search(r'Precision@3:\s+(\d.\d+)', result.output)
+    precision3 = re.search(r"Precision@3:\s+(\d.\d+)", result.output)
     assert float(precision3.group(1)) == 0.5
-    precision5 = re.search(r'Precision@5:\s+(\d.\d+)', result.output)
+    precision5 = re.search(r"Precision@5:\s+(\d.\d+)", result.output)
     assert float(precision5.group(1)) == 0.5
-    lrap = re.search(r'LRAP:\s+(\d.\d+)', result.output)
+    lrap = re.search(r"LRAP:\s+(\d.\d+)", result.output)
     assert float(lrap.group(1)) == 0.75
-    true_positives = re.search(r'True positives:\s+(\d+)', result.output)
+    true_positives = re.search(r"True positives:\s+(\d+)", result.output)
     assert int(true_positives.group(1)) == 1
-    false_positives = re.search(r'False positives:\s+(\d+)', result.output)
+    false_positives = re.search(r"False positives:\s+(\d+)", result.output)
     assert int(false_positives.group(1)) == 1
-    false_negatives = re.search(r'False negatives:\s+(\d+)', result.output)
+    false_negatives = re.search(r"False negatives:\s+(\d+)", result.output)
     assert int(false_negatives.group(1)) == 1
-    ndocs = re.search(r'Documents evaluated:\s+(\d+)', result.output)
+    ndocs = re.search(r"Documents evaluated:\s+(\d+)", result.output)
     assert int(ndocs.group(1)) == 2
 
 
 def test_eval_param(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
 
     result = runner.invoke(
-        annif.cli.cli, [
-            'eval', '--backend-param', 'dummy.score=0.0', 'dummy-en',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["eval", "--backend-param", "dummy.score=0.0", "dummy-en", str(tmpdir)],
+    )
     assert not result.exception
     assert result.exit_code == 0
 
     # since zero scores were set with the parameter, there should be no hits
     # at all
-    recall = re.search(r'Recall .*doc.*:\s+(\d.\d+)', result.output)
+    recall = re.search(r"Recall .*doc.*:\s+(\d.\d+)", result.output)
     assert float(recall.group(1)) == 0.0
 
 
 def test_eval_metric(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
     result = runner.invoke(
-        annif.cli.cli, [
-            'eval', '--metric', 'F1@5', '-m', 'NDCG', 'dummy-en',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["eval", "--metric", "F1@5", "-m", "NDCG", "dummy-en", str(tmpdir)],
+    )
     assert not result.exception
     assert result.exit_code == 0
 
-    f1 = re.search(r'F1@5\s*:\s+(\d.\d+)', result.output)
+    f1 = re.search(r"F1@5\s*:\s+(\d.\d+)", result.output)
     assert float(f1.group(1)) > 0.0
-    ndcg = re.search(r'NDCG\s*:\s+(\d.\d+)', result.output)
+    ndcg = re.search(r"NDCG\s*:\s+(\d.\d+)", result.output)
     assert float(ndcg.group(1)) > 0.0
     # check that we only have 2 metrics + "Documents evaluated"
-    assert len(result.output.strip().split('\n')) == 3
+    assert len(result.output.strip().split("\n")) == 3
 
 
 def test_eval_metricsfile(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
-    metricsfile = tmpdir.join('metrics.json')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
+    metricsfile = tmpdir.join("metrics.json")
     result = runner.invoke(
-        annif.cli.cli, [
-            'eval', '--metrics-file', str(metricsfile), 'dummy-en',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["eval", "--metrics-file", str(metricsfile), "dummy-en", str(tmpdir)],
+    )
     assert not result.exception
     assert result.exit_code == 0
 
     metrics = json.load(metricsfile)
-    assert 'F1@5' in metrics
-    assert metrics['F1@5'] > 0.0
-    assert 'NDCG' in metrics
-    assert metrics['NDCG'] > 0.0
-    assert 'Precision_doc_avg' in metrics
-    assert metrics['Precision_doc_avg'] > 0.0
+    assert "F1@5" in metrics
+    assert metrics["F1@5"] > 0.0
+    assert "NDCG" in metrics
+    assert metrics["NDCG"] > 0.0
+    assert "Precision_doc_avg" in metrics
+    assert metrics["Precision_doc_avg"] > 0.0
 
 
 def test_eval_resultsfile(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
-    resultfile = tmpdir.join('results.tsv')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
+    resultfile = tmpdir.join("results.tsv")
     result = runner.invoke(
-        annif.cli.cli, [
-            'eval', '--results-file', str(resultfile), 'dummy-en',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["eval", "--results-file", str(resultfile), "dummy-en", str(tmpdir)],
+    )
     assert not result.exception
     assert result.exit_code == 0
 
     # subject average should equal average of all subject scores in outputfile
     precision = float(
-        re.search(r'Precision .*subj.*:\s+(\d.\d+)', result.output).group(1))
-    recall = float(
-        re.search(r'Recall .*subj.*:\s+(\d.\d+)', result.output).group(1))
+        re.search(r"Precision .*subj.*:\s+(\d.\d+)", result.output).group(1)
+    )
+    recall = float(re.search(r"Recall .*subj.*:\s+(\d.\d+)", result.output).group(1))
     f_measure = float(
-        re.search(r'F1 score .*subj.*:\s+(\d.\d+)', result.output).group(1))
+        re.search(r"F1 score .*subj.*:\s+(\d.\d+)", result.output).group(1)
+    )
     precision_numerator = 0
     recall_numerator = 0
     f_measure_numerator = 0
     denominator = 0
     with resultfile.open() as f:
         header = next(f)
-        assert header.strip('\n') == '\t'.join(['URI',
-                                                'Label',
-                                                'Support',
-                                                'True_positives',
-                                                'False_positives',
-                                                'False_negatives',
-                                                'Precision',
-                                                'Recall',
-                                                'F1_score'])
+        assert header.strip("\n") == "\t".join(
+            [
+                "URI",
+                "Label",
+                "Support",
+                "True_positives",
+                "False_positives",
+                "False_negatives",
+                "Precision",
+                "Recall",
+                "F1_score",
+            ]
+        )
         for line in f:
-            assert line.strip() != ''
-            parts = line.split('\t')
-            if parts[1] == 'dummy':
+            assert line.strip() != ""
+            parts = line.split("\t")
+            if parts[1] == "dummy":
                 assert int(parts[2]) == 1
                 assert int(parts[3]) == 1
                 assert int(parts[4]) == 1
                 assert int(parts[5]) == 0
-            if parts[1] == 'none':
+            if parts[1] == "none":
                 assert int(parts[2]) == 1
                 assert int(parts[3]) == 0
                 assert int(parts[4]) == 0
@@ -758,188 +729,181 @@ def test_eval_resultsfile(tmpdir):
 
 
 def test_eval_badresultsfile(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'eval', '--results-file', 'newdir/test_file.txt', 'dummy-en',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["eval", "--results-file", "newdir/test_file.txt", "dummy-en", str(tmpdir)],
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert 'cannot open results-file for writing' in failed_result.output
+    assert "cannot open results-file for writing" in failed_result.output
 
 
 def test_eval_docfile():
     docfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'documents.tsv')
-    result = runner.invoke(annif.cli.cli, ['eval', 'dummy-fi', docfile])
+        os.path.dirname(__file__), "corpora", "archaeology", "documents.tsv"
+    )
+    result = runner.invoke(annif.cli.cli, ["eval", "dummy-fi", docfile])
     assert not result.exception
     assert result.exit_code == 0
 
 
 def test_eval_empty_file(tmpdir):
-    empty_file = tmpdir.ensure('empty.tsv')
-    failed_result = runner.invoke(
-        annif.cli.cli, [
-            'eval', 'dummy-fi', str(empty_file)])
+    empty_file = tmpdir.ensure("empty.tsv")
+    failed_result = runner.invoke(annif.cli.cli, ["eval", "dummy-fi", str(empty_file)])
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert 'cannot evaluate empty corpus' in failed_result.output
+    assert "cannot evaluate empty corpus" in failed_result.output
 
 
 def test_eval_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'eval', 'dummy-fi', 'nonexistent_path'])
+        annif.cli.cli, ["eval", "dummy-fi", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for '[PATHS]...': " \
-           "Path 'nonexistent_path' does not exist." in failed_result.output
+    assert (
+        "Invalid value for '[PATHS]...': "
+        "Path 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_eval_single_process(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
 
     result = runner.invoke(
-        annif.cli.cli, ['eval', '--jobs', '1', 'dummy-en', str(tmpdir)])
+        annif.cli.cli, ["eval", "--jobs", "1", "dummy-en", str(tmpdir)]
+    )
     assert not result.exception
     assert result.exit_code == 0
 
 
 def test_eval_two_jobs(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
 
     result = runner.invoke(
-        annif.cli.cli, ['eval', '--jobs', '2', 'dummy-en', str(tmpdir)])
+        annif.cli.cli, ["eval", "--jobs", "2", "dummy-en", str(tmpdir)]
+    )
     assert not result.exception
     assert result.exit_code == 0
 
 
 def test_optimize_dir(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    tmpdir.join("doc3.txt").write("doc3")
 
-    result = runner.invoke(
-        annif.cli.cli, [
-            'optimize', 'dummy-en', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["optimize", "dummy-en", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
 
-    precision = re.search(r'Best\s+Precision .*?doc.*?:\s+(\d.\d+)',
-                          result.output)
+    precision = re.search(r"Best\s+Precision .*?doc.*?:\s+(\d.\d+)", result.output)
     assert float(precision.group(1)) == 0.5
-    recall = re.search(r'Best\s+Recall .*?doc.*?:\s+(\d.\d+)', result.output)
+    recall = re.search(r"Best\s+Recall .*?doc.*?:\s+(\d.\d+)", result.output)
     assert float(recall.group(1)) == 0.5
-    f_measure = re.search(r'Best\s+F1 score .*?doc.*?:\s+(\d.\d+)',
-                          result.output)
+    f_measure = re.search(r"Best\s+F1 score .*?doc.*?:\s+(\d.\d+)", result.output)
     assert float(f_measure.group(1)) == 0.5
-    ndocs = re.search(r'Documents evaluated:\s+(\d)', result.output)
+    ndocs = re.search(r"Documents evaluated:\s+(\d)", result.output)
     assert int(ndocs.group(1)) == 2
 
 
 def test_optimize_docfile(tmpdir):
-    docfile = tmpdir.join('documents.tsv')
-    docfile.write("""Läntinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents.tsv")
+    docfile.write(
+        """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
-        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+    )
 
-    result = runner.invoke(
-        annif.cli.cli, [
-            'optimize', 'dummy-fi', str(docfile)])
+    result = runner.invoke(annif.cli.cli, ["optimize", "dummy-fi", str(docfile)])
     assert not result.exception
     assert result.exit_code == 0
 
 
 def test_optimize_nonexistent_path():
     failed_result = runner.invoke(
-        annif.cli.cli, [
-            'optimize', 'dummy-fi', 'nonexistent_path'])
+        annif.cli.cli, ["optimize", "dummy-fi", "nonexistent_path"]
+    )
     assert failed_result.exception
     assert failed_result.exit_code != 0
-    assert "Invalid value for '[PATHS]...': " \
-           "Path 'nonexistent_path' does not exist." in failed_result.output
+    assert (
+        "Invalid value for '[PATHS]...': "
+        "Path 'nonexistent_path' does not exist." in failed_result.output
+    )
 
 
 def test_hyperopt_ensemble(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
 
-    result = runner.invoke(
-        annif.cli.cli, [
-            'hyperopt', 'ensemble', str(tmpdir)])
+    result = runner.invoke(annif.cli.cli, ["hyperopt", "ensemble", str(tmpdir)])
     assert not result.exception
     assert result.exit_code == 0
 
-    assert re.search(
-        r'sources=dummy-en:0.\d+,dummy-private:0.\d+',
-        result.output) is not None
+    assert (
+        re.search(r"sources=dummy-en:0.\d+,dummy-private:0.\d+", result.output)
+        is not None
+    )
 
 
 def test_hyperopt_ensemble_resultsfile(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
-    resultfile = tmpdir.join('results.tsv')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
+    resultfile = tmpdir.join("results.tsv")
 
     result = runner.invoke(
-        annif.cli.cli, [
-            'hyperopt', '--results-file', str(resultfile), 'ensemble',
-            str(tmpdir)])
+        annif.cli.cli,
+        ["hyperopt", "--results-file", str(resultfile), "ensemble", str(tmpdir)],
+    )
     assert not result.exception
     assert result.exit_code == 0
 
     with resultfile.open() as f:
         header = next(f)
-        assert header.strip('\n') == '\t'.join(['trial',
-                                                'value',
-                                                'dummy-en',
-                                                'dummy-private'])
+        assert header.strip("\n") == "\t".join(
+            ["trial", "value", "dummy-en", "dummy-private"]
+        )
         for idx, line in enumerate(f):
-            assert line.strip() != ''
-            parts = line.split('\t')
+            assert line.strip() != ""
+            parts = line.split("\t")
             assert len(parts) == 4
             assert int(parts[0]) == idx
 
 
 def test_hyperopt_not_supported(tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('dummy')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('none')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("dummy")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("none")
 
-    failed_result = runner.invoke(
-        annif.cli.cli, [
-            'hyperopt', 'tfidf-en', str(tmpdir)])
+    failed_result = runner.invoke(annif.cli.cli, ["hyperopt", "tfidf-en", str(tmpdir)])
     assert failed_result.exception
     assert failed_result.exit_code != 0
 
-    assert 'Hyperparameter optimization not supported' in failed_result.output
+    assert "Hyperparameter optimization not supported" in failed_result.output
 
 
 def test_version_option():
-    result = runner.invoke(
-        annif.cli.cli, ['--version'])
+    result = runner.invoke(annif.cli.cli, ["--version"])
     assert not result.exception
     assert result.exit_code == 0
-    version = importlib.metadata.version('annif')
+    version = importlib.metadata.version("annif")
     assert result.output.strip() == version.strip()
diff --git a/tests/test_config.py b/tests/test_config.py
index 6226afb32..d4b4d3b64 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -7,87 +7,89 @@
 
 
 def test_check_config_exists_cfg():
-    cfg = annif.config.check_config('tests/projects.cfg')
-    assert cfg == 'tests/projects.cfg'
+    cfg = annif.config.check_config("tests/projects.cfg")
+    assert cfg == "tests/projects.cfg"
 
 
 def test_check_config_exists_toml():
-    cfg = annif.config.check_config('tests/projects.toml')
-    assert cfg == 'tests/projects.toml'
+    cfg = annif.config.check_config("tests/projects.toml")
+    assert cfg == "tests/projects.toml"
 
 
 def test_check_config_exists_directory():
-    cfg = annif.config.check_config('tests/projects.d')
-    assert cfg == 'tests/projects.d'
+    cfg = annif.config.check_config("tests/projects.d")
+    assert cfg == "tests/projects.d"
 
 
 def test_check_config_not_exists(caplog):
     with caplog.at_level(logging.WARNING):
-        cfg = annif.config.check_config('tests/notfound.cfg')
+        cfg = annif.config.check_config("tests/notfound.cfg")
     assert cfg is None
-    assert 'Project configuration file or directory "tests/notfound.cfg" ' + \
-        'is missing' in caplog.text
+    assert (
+        'Project configuration file or directory "tests/notfound.cfg" ' + "is missing"
+        in caplog.text
+    )
 
 
 def test_find_config_exists_default(monkeypatch):
     # temporarily chdir to the tests directory
-    monkeypatch.chdir('tests')
+    monkeypatch.chdir("tests")
     cfg = annif.config.find_config()
-    assert cfg == 'projects.cfg'
+    assert cfg == "projects.cfg"
 
 
 def test_find_config_not_exists_default(monkeypatch, caplog):
     # temporarily chdir to the tests/corpora directory
-    monkeypatch.chdir('tests/corpora')
+    monkeypatch.chdir("tests/corpora")
     with caplog.at_level(logging.WARNING):
         cfg = annif.config.find_config()
     assert cfg is None
-    assert 'Could not find project configuration ' in caplog.text
+    assert "Could not find project configuration " in caplog.text
 
 
 def test_parse_config_cfg_nondefault():
-    cfg = annif.config.parse_config('tests/projects.cfg')
+    cfg = annif.config.parse_config("tests/projects.cfg")
     assert isinstance(cfg, annif.config.AnnifConfigCFG)
     assert len(cfg.project_ids) == 17
-    assert cfg['dummy-fi'] is not None
+    assert cfg["dummy-fi"] is not None
 
 
 def test_parse_config_cfg_default(monkeypatch):
     # temporarily chdir to the tests directory
-    monkeypatch.chdir('tests')
-    cfg = annif.config.parse_config('')
+    monkeypatch.chdir("tests")
+    cfg = annif.config.parse_config("")
     assert isinstance(cfg, annif.config.AnnifConfigCFG)
     assert len(cfg.project_ids) == 17
-    assert cfg['dummy-fi'] is not None
+    assert cfg["dummy-fi"] is not None
 
 
 def test_parse_config_toml():
-    cfg = annif.config.parse_config('tests/projects.toml')
+    cfg = annif.config.parse_config("tests/projects.toml")
     assert isinstance(cfg, annif.config.AnnifConfigTOML)
     assert len(cfg.project_ids) == 2
-    assert cfg['dummy-fi-toml'] is not None
+    assert cfg["dummy-fi-toml"] is not None
 
 
 def test_parse_config_toml_failed(tmpdir):
-    conffile = tmpdir.join('projects.toml')
+    conffile = tmpdir.join("projects.toml")
     conffile.write("[section]\nkey=unquoted\n")
     with pytest.raises(ConfigurationException) as excinfo:
         annif.config.parse_config(str(conffile))
-    assert 'Invalid value' in str(excinfo.value)
+    assert "Invalid value" in str(excinfo.value)
 
 
 def test_parse_config_directory():
-    cfg = annif.config.parse_config('tests/projects.d')
+    cfg = annif.config.parse_config("tests/projects.d")
     assert isinstance(cfg, annif.config.AnnifConfigDirectory)
     assert len(cfg.project_ids) == 17 + 2  # projects.cfg + projects.toml
-    assert cfg['dummy-fi'] is not None
-    assert cfg['dummy-fi-toml'] is not None
+    assert cfg["dummy-fi"] is not None
+    assert cfg["dummy-fi-toml"] is not None
 
 
 def test_parse_config_directory_duplicated_project(tmpdir):
-    conffile = tmpdir.join('projects-1.toml')
+    conffile = tmpdir.join("projects-1.toml")
     conffile.write("[duplicated]\nkey='value'\n")
-    conffile = tmpdir.join('projects-2.cfg')
+    conffile = tmpdir.join("projects-2.cfg")
     conffile.write("[duplicated]\nkey=value\n")
     with pytest.raises(ConfigurationException) as excinfo:
         annif.config.parse_config(str(tmpdir))
diff --git a/tests/test_corpus.py b/tests/test_corpus.py
index 264159caf..fd359f015 100644
--- a/tests/test_corpus.py
+++ b/tests/test_corpus.py
@@ -12,7 +12,7 @@ def test_subjectset_uris(subject_index):
     <http://www.yso.fi/onto/yso/p12738>\tviikinkiaika
     """
 
-    sset = annif.corpus.SubjectSet.from_string(data, subject_index, 'fi')
+    sset = annif.corpus.SubjectSet.from_string(data, subject_index, "fi")
     assert len(sset) == 2
     assert subject_index.by_uri("http://www.yso.fi/onto/yso/p2558") in sset
     assert subject_index.by_uri("http://www.yso.fi/onto/yso/p12738") in sset
@@ -23,15 +23,14 @@ def test_subjectset_labels(subject_index):
     viikinkiaika
     """
 
-    sset = annif.corpus.SubjectSet.from_string(data, subject_index, 'fi')
+    sset = annif.corpus.SubjectSet.from_string(data, subject_index, "fi")
     assert len(sset) == 2
-    assert subject_index.by_label("rautakausi", 'fi') in sset
-    assert subject_index.by_label("viikinkiaika", 'fi') in sset
+    assert subject_index.by_label("rautakausi", "fi") in sset
+    assert subject_index.by_label("viikinkiaika", "fi") in sset
 
 
 def test_subjectset_from_list(subject_index):
-    uris = ['http://www.yso.fi/onto/yso/p10849',
-            'http://www.yso.fi/onto/yso/p19740']
+    uris = ["http://www.yso.fi/onto/yso/p10849", "http://www.yso.fi/onto/yso/p19740"]
     subject_ids = [subject_index.by_uri(uri) for uri in uris]
     sset = annif.corpus.SubjectSet(subject_ids)
     assert len(sset) == 2
@@ -62,7 +61,7 @@ def test_subjectset_nonequal():
 
 
 def test_subjectset_as_vector(subject_index):
-    uris = ['http://www.yso.fi/onto/yso/p10849', 'http://example.org/unknown']
+    uris = ["http://www.yso.fi/onto/yso/p10849", "http://example.org/unknown"]
     subject_ids = [subject_index.by_uri(uri) for uri in uris]
     sset = annif.corpus.SubjectSet(subject_ids)
     vector = sset.as_vector(len(subject_index))
@@ -70,7 +69,7 @@ def test_subjectset_as_vector(subject_index):
 
 
 def test_subjectset_as_vector_destination(subject_index):
-    uris = ['http://www.yso.fi/onto/yso/p10849', 'http://example.org/unknown']
+    uris = ["http://www.yso.fi/onto/yso/p10849", "http://example.org/unknown"]
     subject_ids = [subject_index.by_uri(uri) for uri in uris]
     sset = annif.corpus.SubjectSet(subject_ids)
     destination = np.zeros(len(subject_index), dtype=np.float32)
@@ -80,175 +79,187 @@ def test_subjectset_as_vector_destination(subject_index):
 
 
 def test_docdir_key(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('key1')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('key2')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("key1")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("key2")
+    tmpdir.join("doc3.txt").write("doc3")
 
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, 'en')
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, "en")
     files = sorted(list(docdir))
     assert len(files) == 3
-    assert files[0][0] == str(tmpdir.join('doc1.txt'))
-    assert files[0][1] == str(tmpdir.join('doc1.key'))
-    assert files[1][0] == str(tmpdir.join('doc2.txt'))
-    assert files[1][1] == str(tmpdir.join('doc2.key'))
-    assert files[2][0] == str(tmpdir.join('doc3.txt'))
+    assert files[0][0] == str(tmpdir.join("doc1.txt"))
+    assert files[0][1] == str(tmpdir.join("doc1.key"))
+    assert files[1][0] == str(tmpdir.join("doc2.txt"))
+    assert files[1][1] == str(tmpdir.join("doc2.key"))
+    assert files[2][0] == str(tmpdir.join("doc3.txt"))
     assert files[2][1] is None
 
 
 def test_docdir_tsv(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write('<http://example.org/key1>\tkey1')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write('<http://example.org/key2>\tkey2')
-    tmpdir.join('doc3.txt').write('doc3')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("<http://example.org/key1>\tkey1")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("<http://example.org/key2>\tkey2")
+    tmpdir.join("doc3.txt").write("doc3")
 
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, 'en')
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, "en")
     files = sorted(list(docdir))
     assert len(files) == 3
-    assert files[0][0] == str(tmpdir.join('doc1.txt'))
-    assert files[0][1] == str(tmpdir.join('doc1.tsv'))
-    assert files[1][0] == str(tmpdir.join('doc2.txt'))
-    assert files[1][1] == str(tmpdir.join('doc2.tsv'))
-    assert files[2][0] == str(tmpdir.join('doc3.txt'))
+    assert files[0][0] == str(tmpdir.join("doc1.txt"))
+    assert files[0][1] == str(tmpdir.join("doc1.tsv"))
+    assert files[1][0] == str(tmpdir.join("doc2.txt"))
+    assert files[1][1] == str(tmpdir.join("doc2.tsv"))
+    assert files[2][0] == str(tmpdir.join("doc3.txt"))
     assert files[2][1] is None
 
 
 def test_docdir_tsv_bom(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1'.encode('utf-8-sig'))
-    tmpdir.join('doc1.tsv').write(
-        '<http://www.yso.fi/onto/yso/p4622>\tesihistoria'.encode('utf-8-sig'))
-    tmpdir.join('doc2.txt').write('doc2'.encode('utf-8-sig'))
-    tmpdir.join('doc2.tsv').write(
-        '<http://www.yso.fi/onto/yso/p2558>\trautakausi'.encode('utf-8-sig'))
-
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, 'fi')
+    tmpdir.join("doc1.txt").write("doc1".encode("utf-8-sig"))
+    tmpdir.join("doc1.tsv").write(
+        "<http://www.yso.fi/onto/yso/p4622>\tesihistoria".encode("utf-8-sig")
+    )
+    tmpdir.join("doc2.txt").write("doc2".encode("utf-8-sig"))
+    tmpdir.join("doc2.tsv").write(
+        "<http://www.yso.fi/onto/yso/p2558>\trautakausi".encode("utf-8-sig")
+    )
+
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index, "fi")
     docs = list(docdir.documents)
-    assert docs[0].text == 'doc1'
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p4622') in docs[0].subject_set
+    assert docs[0].text == "doc1"
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p4622") in docs[0].subject_set
+    )
     assert len(docs[0].subject_set) == 1
-    assert docs[1].text == 'doc2'
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p2558') in docs[1].subject_set
+    assert docs[1].text == "doc2"
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p2558") in docs[1].subject_set
+    )
     assert len(docs[1].subject_set) == 1
 
 
 def test_docdir_key_require_subjects(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('<http://example.org/key1>\tkey1')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('<http://example.org/key2>\tkey2')
-    tmpdir.join('doc3.txt').write('doc3')
-
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index,
-                                            'en', require_subjects=True)
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("<http://example.org/key1>\tkey1")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("<http://example.org/key2>\tkey2")
+    tmpdir.join("doc3.txt").write("doc3")
+
+    docdir = annif.corpus.DocumentDirectory(
+        str(tmpdir), subject_index, "en", require_subjects=True
+    )
     files = sorted(list(docdir))
     assert len(files) == 2
-    assert files[0][0] == str(tmpdir.join('doc1.txt'))
-    assert files[0][1] == str(tmpdir.join('doc1.key'))
-    assert files[1][0] == str(tmpdir.join('doc2.txt'))
-    assert files[1][1] == str(tmpdir.join('doc2.key'))
+    assert files[0][0] == str(tmpdir.join("doc1.txt"))
+    assert files[0][1] == str(tmpdir.join("doc1.key"))
+    assert files[1][0] == str(tmpdir.join("doc2.txt"))
+    assert files[1][1] == str(tmpdir.join("doc2.key"))
 
 
 def test_docdir_tsv_require_subjects(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write('key1')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write('key2')
-    tmpdir.join('doc3.txt').write('doc3')
-
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index,
-                                            'en', require_subjects=True)
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("key1")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("key2")
+    tmpdir.join("doc3.txt").write("doc3")
+
+    docdir = annif.corpus.DocumentDirectory(
+        str(tmpdir), subject_index, "en", require_subjects=True
+    )
     files = sorted(list(docdir))
     assert len(files) == 2
-    assert files[0][0] == str(tmpdir.join('doc1.txt'))
-    assert files[0][1] == str(tmpdir.join('doc1.tsv'))
-    assert files[1][0] == str(tmpdir.join('doc2.txt'))
-    assert files[1][1] == str(tmpdir.join('doc2.tsv'))
+    assert files[0][0] == str(tmpdir.join("doc1.txt"))
+    assert files[0][1] == str(tmpdir.join("doc1.tsv"))
+    assert files[1][0] == str(tmpdir.join("doc2.txt"))
+    assert files[1][1] == str(tmpdir.join("doc2.tsv"))
 
 
 def test_docdir_tsv_as_doccorpus(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write(
-        '<http://www.yso.fi/onto/yso/p4622>\tesihistoria')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write(
-        '<http://www.yso.fi/onto/yso/p2558>\trautakausi')
-    tmpdir.join('doc3.txt').write('doc3')
-
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index,
-                                            'fi', require_subjects=True)
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("<http://www.yso.fi/onto/yso/p4622>\tesihistoria")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("<http://www.yso.fi/onto/yso/p2558>\trautakausi")
+    tmpdir.join("doc3.txt").write("doc3")
+
+    docdir = annif.corpus.DocumentDirectory(
+        str(tmpdir), subject_index, "fi", require_subjects=True
+    )
     docs = list(docdir.documents)
     assert len(docs) == 2
-    assert docs[0].text == 'doc1'
+    assert docs[0].text == "doc1"
     assert len(docs[0].subject_set) == 1
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p4622') in docs[0].subject_set
-    assert docs[1].text == 'doc2'
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p2558') in docs[1].subject_set
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p4622") in docs[0].subject_set
+    )
+    assert docs[1].text == "doc2"
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p2558") in docs[1].subject_set
+    )
     assert len(docs[1].subject_set) == 1
 
 
 def test_docdir_key_as_doccorpus(tmpdir, subject_index):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.key').write('arkeologit')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.key').write('kalliotaide')
-    tmpdir.join('doc3.txt').write('doc3')
-
-    docdir = annif.corpus.DocumentDirectory(str(tmpdir), subject_index,
-                                            'fi', require_subjects=True)
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.key").write("arkeologit")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.key").write("kalliotaide")
+    tmpdir.join("doc3.txt").write("doc3")
+
+    docdir = annif.corpus.DocumentDirectory(
+        str(tmpdir), subject_index, "fi", require_subjects=True
+    )
     docs = list(docdir.documents)
     assert len(docs) == 2
-    assert docs[0].text == 'doc1'
+    assert docs[0].text == "doc1"
     assert len(docs[0].subject_set) == 1
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p10849') in docs[0].subject_set
-    assert docs[1].text == 'doc2'
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p10849") in docs[0].subject_set
+    )
+    assert docs[1].text == "doc2"
     assert len(docs[1].subject_set) == 1
-    assert subject_index.by_uri(
-        'http://www.yso.fi/onto/yso/p13027') in docs[1].subject_set
+    assert (
+        subject_index.by_uri("http://www.yso.fi/onto/yso/p13027") in docs[1].subject_set
+    )
 
 
 def test_subject_by_uri(subject_index):
-    subj_id = subject_index.by_uri('http://www.yso.fi/onto/yso/p7141')
-    assert subject_index[subj_id].labels['fi'] == 'sinetit'
+    subj_id = subject_index.by_uri("http://www.yso.fi/onto/yso/p7141")
+    assert subject_index[subj_id].labels["fi"] == "sinetit"
 
 
 def test_subject_by_uri_missing(subject_index):
-    subj_id = subject_index.by_uri('http://nonexistent')
+    subj_id = subject_index.by_uri("http://nonexistent")
     assert subj_id is None
 
 
 def test_subject_by_label(subject_index):
-    subj_id = subject_index.by_label('sinetit', 'fi')
-    assert subject_index[subj_id].uri == 'http://www.yso.fi/onto/yso/p7141'
+    subj_id = subject_index.by_label("sinetit", "fi")
+    assert subject_index[subj_id].uri == "http://www.yso.fi/onto/yso/p7141"
 
 
 def test_subject_by_label_missing(subject_index):
-    subj_id = subject_index.by_label('nonexistent', 'fi')
+    subj_id = subject_index.by_label("nonexistent", "fi")
     assert subj_id is None
 
 
 def test_docfile_plain(tmpdir, subject_index):
-    docfile = tmpdir.join('documents.tsv')
-    docfile.write("""Läntinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents.tsv")
+    docfile.write(
+        """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
-        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+    )
 
     docs = annif.corpus.DocumentFile(str(docfile), subject_index)
     assert len(list(docs.documents)) == 3
 
 
 def test_docfile_bom(tmpdir, subject_index):
-    docfile = tmpdir.join('documents_bom.tsv')
+    docfile = tmpdir.join("documents_bom.tsv")
     data = """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
         Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
-    docfile.write(data.encode('utf-8-sig'))
+    docfile.write(data.encode("utf-8-sig"))
 
     docs = annif.corpus.DocumentFile(str(docfile), subject_index)
     firstdoc = next(docs.documents)
@@ -258,42 +269,48 @@ def test_docfile_bom(tmpdir, subject_index):
 def test_docfile_plain_invalid_lines(tmpdir, caplog, subject_index):
     logger = annif.logger
     logger.propagate = True
-    docfile = tmpdir.join('documents_invalid.tsv')
-    docfile.write("""Läntinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents_invalid.tsv")
+    docfile.write(
+        """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
 
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
         A line with no tabs
-        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+    )
     docs = annif.corpus.DocumentFile(str(docfile), subject_index)
     assert len(list(docs.documents)) == 3
     assert len(caplog.records) == 2
-    expected_msg = 'Skipping invalid line (missing tab):'
+    expected_msg = "Skipping invalid line (missing tab):"
     for record in caplog.records:
         assert expected_msg in record.message
 
 
 def test_docfile_gzipped(tmpdir, subject_index):
-    docfile = tmpdir.join('documents.tsv.gz')
-    with gzip.open(str(docfile), 'wt') as gzf:
-        gzf.write("""Pohjoinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents.tsv.gz")
+    with gzip.open(str(docfile), "wt") as gzf:
+        gzf.write(
+            """Pohjoinen\t<http://www.yso.fi/onto/yso/p2557>
             Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
-            Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+            Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+        )
 
     docs = annif.corpus.DocumentFile(str(docfile), subject_index)
     assert len(list(docs.documents)) == 3
 
 
 def test_docfile_is_empty(tmpdir, subject_index):
-    empty_file = tmpdir.ensure('empty.tsv')
+    empty_file = tmpdir.ensure("empty.tsv")
     docs = annif.corpus.DocumentFile(str(empty_file), subject_index)
     assert docs.is_empty()
 
 
 def test_combinedcorpus(tmpdir, subject_index):
-    docfile = tmpdir.join('documents.tsv')
-    docfile.write("""Läntinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents.tsv")
+    docfile.write(
+        """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
-        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+    )
 
     corpus1 = annif.corpus.DocumentFile(str(docfile), subject_index)
     corpus2 = annif.corpus.DocumentFile(str(docfile), subject_index)
@@ -304,30 +321,32 @@ def test_combinedcorpus(tmpdir, subject_index):
 
 
 def test_transformingcorpus(document_corpus):
-    def double(x): return x + x
+    def double(x):
+        return x + x
 
     transformed_corpus = TransformingDocumentCorpus(document_corpus, double)
-    for transf_doc, doc in zip(transformed_corpus.documents,
-                               document_corpus.documents):
+    for transf_doc, doc in zip(transformed_corpus.documents, document_corpus.documents):
         assert transf_doc.text == doc.text + doc.text
         assert transf_doc.subject_set == doc.subject_set
     # Ensure docs are still available after iterating
-    assert len(list(transformed_corpus.documents)) \
-        == len(list(document_corpus.documents))
+    assert len(list(transformed_corpus.documents)) == len(
+        list(document_corpus.documents)
+    )
 
 
 def test_limitingcorpus(tmpdir, subject_index):
-    docfile = tmpdir.join('documents_invalid.tsv')
-    docfile.write("""Läntinen\t<http://www.yso.fi/onto/yso/p2557>
+    docfile = tmpdir.join("documents_invalid.tsv")
+    docfile.write(
+        """Läntinen\t<http://www.yso.fi/onto/yso/p2557>
 
         Oulunlinnan\t<http://www.yso.fi/onto/yso/p7346>
         A line with no tabs
-        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>""")
+        Harald Hirmuinen\t<http://www.yso.fi/onto/yso/p6479>"""
+    )
 
     document_corpus = annif.corpus.DocumentFile(str(docfile), subject_index)
     limiting_corpus = annif.corpus.LimitingDocumentCorpus(document_corpus, 2)
 
     assert len(list(limiting_corpus.documents)) == 2
-    for limited_doc, doc in zip(limiting_corpus.documents,
-                                document_corpus.documents):
+    for limited_doc, doc in zip(limiting_corpus.documents, document_corpus.documents):
         assert limited_doc.text == doc.text
diff --git a/tests/test_eval.py b/tests/test_eval.py
index ce283be82..ba169b778 100644
--- a/tests/test_eval.py
+++ b/tests/test_eval.py
@@ -97,28 +97,34 @@ def test_evaluation_batch(subject_index):
     batch = annif.eval.EvaluationBatch(subject_index)
 
     gold_set = annif.corpus.SubjectSet.from_string(
-        '<http://www.yso.fi/onto/yso/p10849>\tarkeologit',
-        subject_index, 'fi')
-    hits1 = annif.suggestion.ListSuggestionResult([
-        # subject: archaeologists (yso:p10849)
-        annif.suggestion.SubjectSuggestion(
-            subject_id=subject_index.by_uri(
-                'http://www.yso.fi/onto/yso/p10849'),
-            score=1.0)])
+        "<http://www.yso.fi/onto/yso/p10849>\tarkeologit", subject_index, "fi"
+    )
+    hits1 = annif.suggestion.ListSuggestionResult(
+        [
+            # subject: archaeologists (yso:p10849)
+            annif.suggestion.SubjectSuggestion(
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p10849"),
+                score=1.0,
+            )
+        ]
+    )
     batch.evaluate(hits1, gold_set)
-    hits2 = annif.suggestion.ListSuggestionResult([
-        # subject: egyptologists (yso:p1747)
-        annif.suggestion.SubjectSuggestion(
-            subject_id=subject_index.by_uri(
-                'http://www.yso.fi/onto/yso/p1747'),
-            score=1.0)])
+    hits2 = annif.suggestion.ListSuggestionResult(
+        [
+            # subject: egyptologists (yso:p1747)
+            annif.suggestion.SubjectSuggestion(
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p1747"),
+                score=1.0,
+            )
+        ]
+    )
     batch.evaluate(hits2, gold_set)
     results = batch.results()
-    assert results['Precision (doc avg)'] == 0.5
-    assert results['Recall (doc avg)'] == 0.5
-    assert results['LRAP'] >= 0.50
-    assert results['LRAP'] <= 0.51
-    assert results['True positives'] == 1
-    assert results['False positives'] == 1
-    assert results['False negatives'] == 1
-    assert results['Documents evaluated'] == 2
+    assert results["Precision (doc avg)"] == 0.5
+    assert results["Recall (doc avg)"] == 0.5
+    assert results["LRAP"] >= 0.50
+    assert results["LRAP"] <= 0.51
+    assert results["True positives"] == 1
+    assert results["False positives"] == 1
+    assert results["False negatives"] == 1
+    assert results["Documents evaluated"] == 2
diff --git a/tests/test_lexical_mllm.py b/tests/test_lexical_mllm.py
index 2422380ba..7b81055e7 100644
--- a/tests/test_lexical_mllm.py
+++ b/tests/test_lexical_mllm.py
@@ -9,9 +9,8 @@
 def test_mllmmodel_prepare_terms(vocabulary):
     model = MLLMModel()
     graph = vocabulary.as_graph()
-    params = {'language': 'fi', 'use_hidden_labels': True}
-    terms, subject_ids = model._prepare_terms(
-        graph, vocabulary, params)
+    params = {"language": "fi", "use_hidden_labels": True}
+    terms, subject_ids = model._prepare_terms(graph, vocabulary, params)
     assert len(terms) == 163  # 130 prefLabels + 33 altLabels
     assert len(subject_ids) == 130  # 130 subjects
 
@@ -44,26 +43,22 @@ def test_mllmmodel_prepare_relations(vocabulary):
 
     # check some example cases by looking up their subject IDs
 
-    shipfinds = vocabulary.subjects.by_uri('http://www.yso.fi/onto/yso/p8869')
+    shipfinds = vocabulary.subjects.by_uri("http://www.yso.fi/onto/yso/p8869")
     # "ship finds" is not in any collection
     assert c_matrix[:, shipfinds].sum() == 0
 
-    seals = vocabulary.subjects.by_uri('http://www.yso.fi/onto/yso/p7141')
+    seals = vocabulary.subjects.by_uri("http://www.yso.fi/onto/yso/p7141")
     # "seals" is in 2 collections
     assert c_matrix[:, seals].sum() == 2
 
-    dating = vocabulary.subjects.by_uri('http://www.yso.fi/onto/yso/p7804')
+    dating = vocabulary.subjects.by_uri("http://www.yso.fi/onto/yso/p7804")
     # "dating (age estimation)" is in 3 collections
     assert c_matrix[:, dating].sum() == 3
 
 
 def test_train_sanity_check():
     model = MLLMModel()
-    params = {
-        'min_samples_leaf': 20,
-        'max_leaf_nodes': 1000,
-        'max_samples': 0.9
-    }
+    params = {"min_samples_leaf": 20, "max_leaf_nodes": 1000, "max_samples": 0.9}
     train_x = [(1, 2), (2, 3)]
     train_y = [False, False]
 
diff --git a/tests/test_project.py b/tests/test_project.py
index e5901ec8a..501fc22f1 100644
--- a/tests/test_project.py
+++ b/tests/test_project.py
@@ -12,104 +12,103 @@
 def test_create_project_wrong_access(registry):
     with pytest.raises(ConfigurationException):
         annif.project.AnnifProject(
-            'example',
-            {'name': 'Example', 'language': 'en', 'access': 'invalid'},
-            '.',
-            registry)
+            "example",
+            {"name": "Example", "language": "en", "access": "invalid"},
+            ".",
+            registry,
+        )
 
 
 def test_get_project_en(registry):
-    project = registry.get_project('dummy-en')
-    assert project.project_id == 'dummy-en'
-    assert project.language == 'en'
-    assert project.analyzer.name == 'snowball'
-    assert project.analyzer.param == 'english'
+    project = registry.get_project("dummy-en")
+    assert project.project_id == "dummy-en"
+    assert project.language == "en"
+    assert project.analyzer.name == "snowball"
+    assert project.analyzer.param == "english"
     assert project.access == Access.hidden
     assert isinstance(project.backend, annif.backend.dummy.DummyBackend)
 
 
 def test_get_project_fi(registry):
-    project = registry.get_project('dummy-fi')
-    assert project.project_id == 'dummy-fi'
-    assert project.language == 'fi'
-    assert project.analyzer.name == 'snowball'
-    assert project.analyzer.param == 'finnish'
-    assert project.vocab_lang == 'fi'
+    project = registry.get_project("dummy-fi")
+    assert project.project_id == "dummy-fi"
+    assert project.language == "fi"
+    assert project.analyzer.name == "snowball"
+    assert project.analyzer.param == "finnish"
+    assert project.vocab_lang == "fi"
     assert project.access == Access.public
     assert isinstance(project.backend, annif.backend.dummy.DummyBackend)
 
 
 def test_get_project_dummy_private(registry):
-    project = registry.get_project('dummy-private')
-    assert project.project_id == 'dummy-private'
-    assert project.language == 'en'
-    assert project.analyzer.name == 'snowball'
-    assert project.analyzer.param == 'english'
+    project = registry.get_project("dummy-private")
+    assert project.project_id == "dummy-private"
+    assert project.language == "en"
+    assert project.analyzer.name == "snowball"
+    assert project.analyzer.param == "english"
     assert project.access == Access.private
     assert isinstance(project.backend, annif.backend.dummy.DummyBackend)
 
 
 def test_get_project_dummy_vocablang(registry):
-    project = registry.get_project('dummy-vocablang')
-    assert project.project_id == 'dummy-vocablang'
-    assert project.language == 'en'
-    assert project.analyzer.name == 'snowball'
-    assert project.analyzer.param == 'english'
+    project = registry.get_project("dummy-vocablang")
+    assert project.project_id == "dummy-vocablang"
+    assert project.language == "en"
+    assert project.analyzer.name == "snowball"
+    assert project.analyzer.param == "english"
     # project uses the dummy vocab, with language overridden to Finnish
-    assert project.vocab.vocab_id == 'dummy'
-    assert project.vocab_lang == 'fi'
+    assert project.vocab.vocab_id == "dummy"
+    assert project.vocab_lang == "fi"
     assert project.access == Access.public
     assert isinstance(project.backend, annif.backend.dummy.DummyBackend)
 
 
 def test_get_project_fi_dump(registry):
-    project = registry.get_project('dummy-fi')
+    project = registry.get_project("dummy-fi")
     pdump = project.dump()
     assert pdump == {
-        'project_id': 'dummy-fi',
-        'name': 'Dummy Finnish',
-        'language': 'fi',
-        'backend': {
-            'backend_id': 'dummy',
+        "project_id": "dummy-fi",
+        "name": "Dummy Finnish",
+        "language": "fi",
+        "backend": {
+            "backend_id": "dummy",
         },
-        'is_trained': True,
-        'modification_time': None,
+        "is_trained": True,
+        "modification_time": None,
     }
 
 
 def test_get_project_nonexistent(registry):
     with pytest.raises(ValueError):
-        registry.get_project('nonexistent')
+        registry.get_project("nonexistent")
 
 
 def test_get_project_noanalyzer(registry):
-    project = registry.get_project('noanalyzer')
+    project = registry.get_project("noanalyzer")
     with pytest.raises(ConfigurationException):
         project.analyzer
 
 
 def test_get_project_novocab(registry):
-    project = registry.get_project('novocab')
+    project = registry.get_project("novocab")
     with pytest.raises(ConfigurationException):
         project.vocab
 
 
 def test_get_project_nobackend(registry):
-    project = registry.get_project('nobackend')
+    project = registry.get_project("nobackend")
     with pytest.raises(ConfigurationException):
         project.backend
 
 
 def test_get_project_noname(registry):
-    project = registry.get_project('noname')
+    project = registry.get_project("noname")
     assert project.name == project.project_id
 
 
 def test_get_project_default_params_tfidf(registry):
-    project = registry.get_project('noparams-tfidf-fi')
-    expected_default_params = {
-        'limit': 100  # From AnnifBackend class
-    }
+    project = registry.get_project("noparams-tfidf-fi")
+    expected_default_params = {"limit": 100}  # From AnnifBackend class
     actual_params = project.backend.params
     for param, val in expected_default_params.items():
         assert param in actual_params and actual_params[param] == val
@@ -117,13 +116,14 @@ def test_get_project_default_params_tfidf(registry):
 
 def test_get_project_default_params_fasttext(registry):
     pytest.importorskip("annif.backend.fasttext")
-    project = registry.get_project('noparams-fasttext-fi')
+    project = registry.get_project("noparams-fasttext-fi")
     expected_default_params = {
-        'limit': 100,  # From AnnifBackend class
-        'dim': 100,    # Rest from FastTextBackend class
-        'lr': 0.25,
-        'epoch': 5,
-        'loss': 'hs'}
+        "limit": 100,  # From AnnifBackend class
+        "dim": 100,  # Rest from FastTextBackend class
+        "lr": 0.25,
+        "epoch": 5,
+        "loss": "hs",
+    }
     actual_params = project.backend.params
     for param, val in expected_default_params.items():
         assert param in actual_params and actual_params[param] == val
@@ -131,162 +131,150 @@ def test_get_project_default_params_fasttext(registry):
 
 def test_get_project_invalid_config_file():
     app = annif.create_app(
-        config_name='annif.default_config.TestingInvalidProjectsConfig')
+        config_name="annif.default_config.TestingInvalidProjectsConfig"
+    )
     with app.app_context():
         with pytest.raises(ConfigurationException):
-            annif.registry.get_project('duplicatedvocab')
+            annif.registry.get_project("duplicatedvocab")
 
 
 def test_project_load_vocabulary_tfidf(registry, subject_file, testdatadir):
-    project = registry.get_project('tfidf-fi')
+    project = registry.get_project("tfidf-fi")
     project.vocab.load_vocabulary(subject_file)
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
 
 
 def test_project_tfidf_is_not_trained(registry):
-    project = registry.get_project('tfidf-fi')
+    project = registry.get_project("tfidf-fi")
     assert not project.is_trained
 
 
 def test_project_train_tfidf(registry, document_corpus, testdatadir):
-    project = registry.get_project('tfidf-fi')
+    project = registry.get_project("tfidf-fi")
     project.train(document_corpus)
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').exists()
-    assert testdatadir.join('projects/tfidf-fi/tfidf-index').size() > 0
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").exists()
+    assert testdatadir.join("projects/tfidf-fi/tfidf-index").size() > 0
 
 
 def test_project_tfidf_is_trained(registry):
-    project = registry.get_project('tfidf-fi')
+    project = registry.get_project("tfidf-fi")
     assert project.is_trained
 
 
 def test_project_tfidf_modification_time(registry):
-    project = registry.get_project('tfidf-fi')
-    assert datetime.now(timezone.utc) - \
-        project.modification_time < timedelta(1)
+    project = registry.get_project("tfidf-fi")
+    assert datetime.now(timezone.utc) - project.modification_time < timedelta(1)
 
 
 def test_project_train_tfidf_nodocuments(registry, empty_corpus):
-    project = registry.get_project('tfidf-fi')
+    project = registry.get_project("tfidf-fi")
     with pytest.raises(NotSupportedException) as excinfo:
         project.train(empty_corpus)
-    assert 'Cannot train tfidf project with no documents' in str(excinfo.value)
+    assert "Cannot train tfidf project with no documents" in str(excinfo.value)
 
 
 def test_project_learn(registry, tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write('<http://example.org/none>\tnone')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write('<http://example.org/dummy>\tdummy')
-
-    project = registry.get_project('dummy-fi')
-    docdir = annif.corpus.DocumentDirectory(
-        str(tmpdir), project.subjects, 'en')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("<http://example.org/none>\tnone")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("<http://example.org/dummy>\tdummy")
+
+    project = registry.get_project("dummy-fi")
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), project.subjects, "en")
     project.learn(docdir)
-    result = project.suggest('this is some text')
+    result = project.suggest("this is some text")
     assert len(result) == 1
     hits = result.as_list()
-    assert hits[0].subject_id == project.subjects.by_uri(
-        'http://example.org/none')
+    assert hits[0].subject_id == project.subjects.by_uri("http://example.org/none")
     assert hits[0].score == 1.0
 
 
 def test_project_learn_not_supported(registry, tmpdir):
-    tmpdir.join('doc1.txt').write('doc1')
-    tmpdir.join('doc1.tsv').write('<http://example.org/key1>\tkey1')
-    tmpdir.join('doc2.txt').write('doc2')
-    tmpdir.join('doc2.tsv').write('<http://example.org/key2>\tkey2')
-
-    project = registry.get_project('tfidf-fi')
-    docdir = annif.corpus.DocumentDirectory(
-        str(tmpdir), project.subjects, 'en')
+    tmpdir.join("doc1.txt").write("doc1")
+    tmpdir.join("doc1.tsv").write("<http://example.org/key1>\tkey1")
+    tmpdir.join("doc2.txt").write("doc2")
+    tmpdir.join("doc2.tsv").write("<http://example.org/key2>\tkey2")
+
+    project = registry.get_project("tfidf-fi")
+    docdir = annif.corpus.DocumentDirectory(str(tmpdir), project.subjects, "en")
     with pytest.raises(NotSupportedException):
         project.learn(docdir)
 
 
 def test_project_load_vocabulary_fasttext(registry, subject_file, testdatadir):
     pytest.importorskip("annif.backend.fasttext")
-    project = registry.get_project('fasttext-fi')
+    project = registry.get_project("fasttext-fi")
     project.vocab.load_vocabulary(subject_file)
-    assert testdatadir.join('vocabs/yso/subjects.csv').exists()
-    assert testdatadir.join('vocabs/yso/subjects.csv').size() > 0
+    assert testdatadir.join("vocabs/yso/subjects.csv").exists()
+    assert testdatadir.join("vocabs/yso/subjects.csv").size() > 0
 
 
 def test_project_train_fasttext(registry, document_corpus, testdatadir):
     pytest.importorskip("annif.backend.fasttext")
-    project = registry.get_project('fasttext-fi')
+    project = registry.get_project("fasttext-fi")
     project.train(document_corpus)
-    assert testdatadir.join('projects/fasttext-fi/fasttext-model').exists()
-    assert testdatadir.join('projects/fasttext-fi/fasttext-model').size() > 0
+    assert testdatadir.join("projects/fasttext-fi/fasttext-model").exists()
+    assert testdatadir.join("projects/fasttext-fi/fasttext-model").size() > 0
 
 
 def test_project_suggest(registry):
-    project = registry.get_project('dummy-en')
-    result = project.suggest('this is some text')
+    project = registry.get_project("dummy-en")
+    result = project.suggest("this is some text")
     assert len(result) == 1
     hits = result.as_list()
-    assert hits[0].subject_id == project.subjects.by_uri(
-        'http://example.org/dummy')
+    assert hits[0].subject_id == project.subjects.by_uri("http://example.org/dummy")
     assert hits[0].score == 1.0
 
 
 def test_project_train_state_not_available(registry, caplog):
-    project = registry.get_project('dummy-vocablang')
+    project = registry.get_project("dummy-vocablang")
     project.backend.is_trained = None
     with caplog.at_level(logging.WARNING):
-        result = project.suggest('this is some text')
+        result = project.suggest("this is some text")
     assert project.is_trained is None
     assert len(result) == 1
     hits = result.as_list()
-    assert hits[0].subject_id == project.subjects.by_uri(
-        'http://example.org/dummy')
+    assert hits[0].subject_id == project.subjects.by_uri("http://example.org/dummy")
     assert hits[0].score == 1.0
-    assert 'Could not get train state information' in caplog.text
+    assert "Could not get train state information" in caplog.text
 
 
 def test_project_transform_text_pass_through(registry):
-    project = registry.get_project('dummy-transform')
-    assert project.transform.transform_text(
-        'this is some text') == 'this is some text'
+    project = registry.get_project("dummy-transform")
+    assert project.transform.transform_text("this is some text") == "this is some text"
 
 
 def test_project_not_initialized(registry):
-    project = registry.get_project('dummy-en')
+    project = registry.get_project("dummy-en")
     assert not project.initialized
 
 
 def test_project_initialized(app_with_initialize):
     with app_with_initialize.app_context():
-        project = annif.registry.get_project('dummy-en')
+        project = annif.registry.get_project("dummy-en")
     assert project.initialized
     assert project.backend.initialized
 
 
 def test_project_file_not_found():
-    app = annif.create_app(
-        config_name='annif.default_config.TestingNoProjectsConfig')
+    app = annif.create_app(config_name="annif.default_config.TestingNoProjectsConfig")
     with app.app_context():
         with pytest.raises(ValueError):
-            annif.registry.get_project('dummy-en')
+            annif.registry.get_project("dummy-en")
 
 
 def test_project_file_toml():
-    app = annif.create_app(
-        config_name='annif.default_config.TestingTOMLConfig')
+    app = annif.create_app(config_name="annif.default_config.TestingTOMLConfig")
     with app.app_context():
         assert len(annif.registry.get_projects()) == 2
-        assert annif.registry.get_project('dummy-fi-toml').project_id \
-            == 'dummy-fi-toml'
-        assert annif.registry.get_project('dummy-en-toml').project_id \
-            == 'dummy-en-toml'
+        assert annif.registry.get_project("dummy-fi-toml").project_id == "dummy-fi-toml"
+        assert annif.registry.get_project("dummy-en-toml").project_id == "dummy-en-toml"
 
 
 def test_project_directory():
-    app = annif.create_app(
-        config_name='annif.default_config.TestingDirectoryConfig')
+    app = annif.create_app(config_name="annif.default_config.TestingDirectoryConfig")
     with app.app_context():
         assert len(annif.registry.get_projects()) == 17 + 2
-        assert annif.registry.get_project('dummy-fi').project_id == 'dummy-fi'
-        assert annif.registry.get_project('dummy-fi-toml').project_id \
-            == 'dummy-fi-toml'
+        assert annif.registry.get_project("dummy-fi").project_id == "dummy-fi"
+        assert annif.registry.get_project("dummy-fi-toml").project_id == "dummy-fi-toml"
diff --git a/tests/test_rest.py b/tests/test_rest.py
index d7d8b6585..b694be1ef 100644
--- a/tests/test_rest.py
+++ b/tests/test_rest.py
@@ -6,41 +6,41 @@
 def test_rest_list_projects(app):
     with app.app_context():
         result = annif.rest.list_projects()
-        project_ids = [proj['project_id'] for proj in result['projects']]
+        project_ids = [proj["project_id"] for proj in result["projects"]]
         # public project should be returned
-        assert 'dummy-fi' in project_ids
+        assert "dummy-fi" in project_ids
         # hidden project should not be returned
-        assert 'dummy-en' not in project_ids
+        assert "dummy-en" not in project_ids
         # private project should not be returned
-        assert 'dummy-private' not in project_ids
+        assert "dummy-private" not in project_ids
         # project with no access level setting should be returned
-        assert 'ensemble' in project_ids
+        assert "ensemble" in project_ids
 
 
 def test_rest_show_project_public(app):
     # public projects should be accessible via REST
     with app.app_context():
-        result = annif.rest.show_project('dummy-fi')
-        assert result['project_id'] == 'dummy-fi'
+        result = annif.rest.show_project("dummy-fi")
+        assert result["project_id"] == "dummy-fi"
 
 
 def test_rest_show_project_hidden(app):
     # hidden projects should be accessible if you know the project id
     with app.app_context():
-        result = annif.rest.show_project('dummy-en')
-        assert result['project_id'] == 'dummy-en'
+        result = annif.rest.show_project("dummy-en")
+        assert result["project_id"] == "dummy-en"
 
 
 def test_rest_show_project_private(app):
     # private projects should not be accessible via REST
     with app.app_context():
-        result = annif.rest.show_project('dummy-private')
+        result = annif.rest.show_project("dummy-private")
         assert result.status_code == 404
 
 
 def test_rest_show_project_nonexistent(app):
     with app.app_context():
-        result = annif.rest.show_project('nonexistent')
+        result = annif.rest.show_project("nonexistent")
         assert result.status_code == 404
 
 
@@ -48,74 +48,66 @@ def test_rest_suggest_public(app):
     # public projects should be accessible via REST
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-fi',
-            text='example text',
-            limit=10,
-            threshold=0.0)
-        assert 'results' in result
+            "dummy-fi", text="example text", limit=10, threshold=0.0
+        )
+        assert "results" in result
 
 
 def test_rest_suggest_hidden(app):
     # hidden projects should be accessible if you know the project id
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-en',
-            text='example text',
-            limit=10,
-            threshold=0.0)
-        assert 'results' in result
+            "dummy-en", text="example text", limit=10, threshold=0.0
+        )
+        assert "results" in result
 
 
 def test_rest_suggest_private(app):
     # private projects should not be accessible via REST
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-private',
-            text='example text',
-            limit=10,
-            threshold=0.0)
+            "dummy-private", text="example text", limit=10, threshold=0.0
+        )
         assert result.status_code == 404
 
 
 def test_rest_suggest_nonexistent(app):
     with app.app_context():
         result = annif.rest.suggest(
-            'nonexistent',
-            text='example text',
-            limit=10,
-            threshold=0.0)
+            "nonexistent", text="example text", limit=10, threshold=0.0
+        )
         assert result.status_code == 404
 
 
 def test_rest_suggest_novocab(app):
     with app.app_context():
         result = annif.rest.suggest(
-            'novocab',
-            text='example text',
-            limit=10,
-            threshold=0.0)
+            "novocab", text="example text", limit=10, threshold=0.0
+        )
         assert result.status_code == 503
 
 
 def test_rest_suggest_with_language_override(app):
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-vocablang',
-            text='example text',
+            "dummy-vocablang",
+            text="example text",
             limit=10,
             threshold=0.0,
-            language='en')
-        assert result['results'][0]['label'] == 'dummy'
+            language="en",
+        )
+        assert result["results"][0]["label"] == "dummy"
 
 
 def test_rest_suggest_with_language_override_bad_value(app):
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-vocablang',
-            text='example text',
+            "dummy-vocablang",
+            text="example text",
             limit=10,
             threshold=0.0,
-            language='xx')
+            language="xx",
+        )
         assert result.status_code == 400
 
 
@@ -124,60 +116,57 @@ def test_rest_suggest_with_different_vocab_language(app):
     # vocab language is Finnish - subject labels should be in Finnish
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-vocablang',
-            text='example text',
-            limit=10,
-            threshold=0.0)
-        assert result['results'][0]['label'] == 'dummy-fi'
+            "dummy-vocablang", text="example text", limit=10, threshold=0.0
+        )
+        assert result["results"][0]["label"] == "dummy-fi"
 
 
 def test_rest_suggest_with_notations(app):
     with app.app_context():
         result = annif.rest.suggest(
-            'dummy-fi',
-            text='example text',
-            limit=10,
-            threshold=0.0)
-        assert result['results'][0]['notation'] is None
+            "dummy-fi", text="example text", limit=10, threshold=0.0
+        )
+        assert result["results"][0]["notation"] is None
 
 
 def test_rest_learn_empty(app):
     with app.app_context():
-        response = annif.rest.learn('dummy-en', [])
+        response = annif.rest.learn("dummy-en", [])
         assert response == (None, 204)  # success, no output
 
 
 def test_rest_learn(app):
-    documents = [{'text': 'the quick brown fox',
-                  'subjects': [{'uri': 'http://example.org/none',
-                                'label': 'none'}]}]
-    with app.app_context():
-        response = annif.rest.learn('dummy-en', documents)
+    documents = [
+        {
+            "text": "the quick brown fox",
+            "subjects": [{"uri": "http://example.org/none", "label": "none"}],
+        }
+    ]
+    with app.app_context():
+        response = annif.rest.learn("dummy-en", documents)
         assert response == (None, 204)  # success, no output
 
         result = annif.rest.suggest(
-            'dummy-en',
-            text='example text',
-            limit=10,
-            threshold=0.0)
-        assert 'results' in result
-        assert result['results'][0]['uri'] == 'http://example.org/none'
-        assert result['results'][0]['label'] == 'none'
+            "dummy-en", text="example text", limit=10, threshold=0.0
+        )
+        assert "results" in result
+        assert result["results"][0]["uri"] == "http://example.org/none"
+        assert result["results"][0]["label"] == "none"
 
 
 def test_rest_learn_novocab(app):
     with app.app_context():
-        result = annif.rest.learn('novocab', [])
+        result = annif.rest.learn("novocab", [])
         assert result.status_code == 503
 
 
 def test_rest_learn_nonexistent(app):
     with app.app_context():
-        result = annif.rest.learn('nonexistent', [])
+        result = annif.rest.learn("nonexistent", [])
         assert result.status_code == 404
 
 
 def test_rest_learn_not_supported(app):
     with app.app_context():
-        result = annif.rest.learn('tfidf-fi', [])
+        result = annif.rest.learn("tfidf-fi", [])
         assert result.status_code == 503
diff --git a/tests/test_suggestion.py b/tests/test_suggestion.py
index eb934bc32..cf40bc2d8 100644
--- a/tests/test_suggestion.py
+++ b/tests/test_suggestion.py
@@ -1,8 +1,13 @@
 """Unit tests for suggestion processing in Annif"""
 
-from annif.suggestion import SubjectSuggestion, SuggestionResult, \
-    LazySuggestionResult, ListSuggestionResult, VectorSuggestionResult, \
-    SuggestionFilter
+from annif.suggestion import (
+    SubjectSuggestion,
+    SuggestionResult,
+    LazySuggestionResult,
+    ListSuggestionResult,
+    VectorSuggestionResult,
+    SuggestionFilter,
+)
 from annif.corpus import Subject
 import numpy as np
 
@@ -10,8 +15,7 @@
 def generate_suggestions(n, subject_index):
     suggestions = []
     for i in range(n):
-        suggestions.append(SubjectSuggestion(subject_id=i,
-                                             score=1.0 / (i + 1)))
+        suggestions.append(SubjectSuggestion(subject_id=i, score=1.0 / (i + 1)))
     return ListSuggestionResult(suggestions)
 
 
@@ -24,42 +28,38 @@ def test_hitfilter_limit(subject_index):
 
 def test_hitfilter_threshold(subject_index):
     origsuggestions = generate_suggestions(10, subject_index)
-    suggestions = SuggestionFilter(subject_index,
-                                   threshold=0.5)(origsuggestions)
+    suggestions = SuggestionFilter(subject_index, threshold=0.5)(origsuggestions)
     assert isinstance(suggestions, SuggestionResult)
     assert len(suggestions) == 2
 
 
 def test_hitfilter_zero_score(subject_index):
-    origsuggestions = ListSuggestionResult(
-        [SubjectSuggestion(subject_id=0,
-                           score=0.0)])
+    origsuggestions = ListSuggestionResult([SubjectSuggestion(subject_id=0, score=0.0)])
     suggestions = SuggestionFilter(subject_index)(origsuggestions)
     assert isinstance(suggestions, SuggestionResult)
     assert len(suggestions) == 0
 
 
-def test_hitfilter_list_suggestion_results_with_deprecated_subjects(
-        subject_index):
-    subject_index.append(Subject(uri='http://example.org/deprecated',
-                                 labels=None,
-                                 notation=None))
+def test_hitfilter_list_suggestion_results_with_deprecated_subjects(subject_index):
+    subject_index.append(
+        Subject(uri="http://example.org/deprecated", labels=None, notation=None)
+    )
     suggestions = ListSuggestionResult(
         [
             # subject: seals (labels)
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p7141'),
-                score=1.0),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p7141"),
+                score=1.0,
+            ),
             # subject: Vikings
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p6479'),
-                score=0.5),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p6479"),
+                score=0.5,
+            ),
             # a deprecated subject
-            SubjectSuggestion(
-                subject_id=None,
-                score=0.5)])
+            SubjectSuggestion(subject_id=None, score=0.5),
+        ]
+    )
     filtered_suggestions = SuggestionFilter(subject_index)(suggestions)
     assert isinstance(filtered_suggestions, SuggestionResult)
     assert len(filtered_suggestions) == 2
@@ -67,19 +67,19 @@ def test_hitfilter_list_suggestion_results_with_deprecated_subjects(
     assert filtered_suggestions.as_list()[1] == suggestions.as_list()[1]
 
 
-def test_hitfilter_vector_suggestion_results_with_deprecated_subjects(
-        subject_index):
-    subject_index.append(Subject(uri='http://example.org/deprecated',
-                                 labels=None,
-                                 notation=None))
+def test_hitfilter_vector_suggestion_results_with_deprecated_subjects(subject_index):
+    subject_index.append(
+        Subject(uri="http://example.org/deprecated", labels=None, notation=None)
+    )
     vector = np.ones(len(subject_index))
     suggestions = VectorSuggestionResult(vector)
     filtered_suggestions = SuggestionFilter(subject_index)(suggestions)
 
-    assert len(suggestions) == len(filtered_suggestions) \
-        + len(subject_index.deprecated_ids())
+    assert len(suggestions) == len(filtered_suggestions) + len(
+        subject_index.deprecated_ids()
+    )
 
-    deprecated_id = subject_index.by_uri('http://example.org/deprecated')
+    deprecated_id = subject_index.by_uri("http://example.org/deprecated")
     deprecated = SubjectSuggestion(subject_id=deprecated_id, score=1.0)
 
     assert deprecated in suggestions.as_list()
@@ -103,14 +103,16 @@ def test_list_suggestion_result_vector(subject_index):
         [
             # subject: seals (labels)
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p7141'),
-                score=1.0),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p7141"),
+                score=1.0,
+            ),
             # subject: Vikings
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p6479'),
-                score=0.5)])
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p6479"),
+                score=0.5,
+            ),
+        ]
+    )
     vector = suggestions.as_vector(len(subject_index))
     assert isinstance(vector, np.ndarray)
     assert len(vector) == len(subject_index)
@@ -118,9 +120,9 @@ def test_list_suggestion_result_vector(subject_index):
     for subject_id, score in enumerate(vector):
         if subject_index[subject_id].labels is None:  # deprecated
             assert score == 0.0
-        elif subject_index[subject_id].labels['fi'] == 'sinetit':
+        elif subject_index[subject_id].labels["fi"] == "sinetit":
             assert score == 1.0
-        elif subject_index[subject_id].labels['fi'] == 'viikingit':
+        elif subject_index[subject_id].labels["fi"] == "viikingit":
             assert score == 0.5
         else:
             assert score == 0.0
@@ -131,39 +133,41 @@ def test_list_suggestions_vector_enforce_score_range(subject_index):
         [
             # subject: seals (labels)
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p7141'),
-                score=1.5),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p7141"),
+                score=1.5,
+            ),
             # subject: Vikings
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p6479'),
-                score=1.0),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p6479"),
+                score=1.0,
+            ),
             # subject: excavations
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p14173'),
-                score=0.5),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p14173"),
+                score=0.5,
+            ),
             # subject: runestones
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p14588'),
-                score=0.0),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p14588"),
+                score=0.0,
+            ),
             # subject: Viking Age
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p12738'),
-                score=-0.5)])
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p12738"),
+                score=-0.5,
+            ),
+        ]
+    )
     vector = suggestions.as_vector(len(subject_index))
     assert vector.sum() == 2.5
     found = 0
     for subject_id, score in enumerate(vector):
         if subject_index[subject_id].labels is None:
             continue  # skip deprecated subjects
-        if subject_index[subject_id].labels['fi'] == 'sinetit':
+        if subject_index[subject_id].labels["fi"] == "sinetit":
             assert score == 1.0
             found += 1
-        elif subject_index[subject_id].labels['fi'] == 'viikinkiaika':
+        elif subject_index[subject_id].labels["fi"] == "viikinkiaika":
             assert score == 0.0
             found += 1
         else:
@@ -176,17 +180,18 @@ def test_list_suggestion_result_vector_destination(subject_index):
         [
             # subject: seals (labels)
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p7141'),
-                score=1.0),
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p7141"),
+                score=1.0,
+            ),
             # subject: Vikings
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://www.yso.fi/onto/yso/p6479'),
-                score=0.5)])
+                subject_id=subject_index.by_uri("http://www.yso.fi/onto/yso/p6479"),
+                score=0.5,
+            ),
+        ]
+    )
     destination = np.zeros(len(subject_index), dtype=np.float32)
-    vector = suggestions.as_vector(len(subject_index),
-                                   destination=destination)
+    vector = suggestions.as_vector(len(subject_index), destination=destination)
     assert vector is destination
 
 
@@ -194,9 +199,11 @@ def test_list_suggestion_result_vector_notfound(subject_index):
     suggestions = ListSuggestionResult(
         [
             SubjectSuggestion(
-                subject_id=subject_index.by_uri(
-                    'http://example.com/notfound'),
-                score=1.0)])
+                subject_id=subject_index.by_uri("http://example.com/notfound"),
+                score=1.0,
+            )
+        ]
+    )
     assert suggestions.as_vector(len(subject_index)).sum() == 0
 
 
@@ -221,7 +228,6 @@ def test_vector_suggestion_result_as_vector_destination(subject_index):
     destination = np.zeros(len(subject_index), dtype=np.float32)
     assert not (destination == orig_vector).all()  # destination is all zeros
 
-    vector = suggestions.as_vector(len(subject_index),
-                                   destination=destination)
+    vector = suggestions.as_vector(len(subject_index), destination=destination)
     assert vector is destination
-    assert (destination == orig_vector).all()      # destination now all ones
+    assert (destination == orig_vector).all()  # destination now all ones
diff --git a/tests/test_swagger.py b/tests/test_swagger.py
index 5a0f87176..037d17195 100644
--- a/tests/test_swagger.py
+++ b/tests/test_swagger.py
@@ -3,66 +3,72 @@
 
 def test_swagger_cors(app_client):
     # test that the service supports CORS
-    req = app_client.get('http://localhost:8000/v1/projects')
-    assert req.headers['access-control-allow-origin'] == '*'
+    req = app_client.get("http://localhost:8000/v1/projects")
+    assert req.headers["access-control-allow-origin"] == "*"
 
 
 def test_swagger_list_projects(app_client):
-    req = app_client.get('http://localhost:8000/v1/projects')
+    req = app_client.get("http://localhost:8000/v1/projects")
     assert req.status_code == 200
-    assert 'projects' in req.get_json()
+    assert "projects" in req.get_json()
 
 
 def test_swagger_show_project(app_client):
-    req = app_client.get('http://localhost:8000/v1/projects/dummy-fi')
+    req = app_client.get("http://localhost:8000/v1/projects/dummy-fi")
     assert req.status_code == 200
-    assert req.get_json()['project_id'] == 'dummy-fi'
+    assert req.get_json()["project_id"] == "dummy-fi"
 
 
 def test_swagger_show_project_nonexistent(app_client):
-    req = app_client.get('http://localhost:8000/v1/projects/nonexistent')
+    req = app_client.get("http://localhost:8000/v1/projects/nonexistent")
     assert req.status_code == 404
 
 
 def test_swagger_suggest(app_client):
-    data = {'text': 'example text'}
+    data = {"text": "example text"}
     req = app_client.post(
-        'http://localhost:8000/v1/projects/dummy-fi/suggest', data=data)
+        "http://localhost:8000/v1/projects/dummy-fi/suggest", data=data
+    )
     assert req.status_code == 200
-    assert 'results' in req.get_json()
+    assert "results" in req.get_json()
 
 
 def test_swagger_suggest_nonexistent(app_client):
-    data = {'text': 'example text'}
+    data = {"text": "example text"}
     req = app_client.post(
-        'http://localhost:8000/v1/projects/nonexistent/suggest', data=data)
+        "http://localhost:8000/v1/projects/nonexistent/suggest", data=data
+    )
     assert req.status_code == 404
 
 
 def test_swagger_suggest_novocab(app_client):
-    data = {'text': 'example text'}
+    data = {"text": "example text"}
     req = app_client.post(
-        'http://localhost:8000/v1/projects/novocab/suggest', data=data)
+        "http://localhost:8000/v1/projects/novocab/suggest", data=data
+    )
     assert req.status_code == 503
 
 
 def test_swagger_learn(app_client):
-    data = [{'text': 'the quick brown fox',
-            'subjects': [{'uri': 'http://example.org/fox', 'label': 'fox'}]}]
-    req = app_client.post(
-        'http://localhost:8000/v1/projects/dummy-fi/learn', json=data)
+    data = [
+        {
+            "text": "the quick brown fox",
+            "subjects": [{"uri": "http://example.org/fox", "label": "fox"}],
+        }
+    ]
+    req = app_client.post("http://localhost:8000/v1/projects/dummy-fi/learn", json=data)
     assert req.status_code == 204
 
 
 def test_swagger_learn_nonexistent(app_client):
     data = []
     req = app_client.post(
-        'http://localhost:8000/v1/projects/nonexistent/learn', json=data)
+        "http://localhost:8000/v1/projects/nonexistent/learn", json=data
+    )
     assert req.status_code == 404
 
 
 def test_swagger_learn_novocab(app_client):
     data = []
-    req = app_client.post(
-        'http://localhost:8000/v1/projects/novocab/learn', json=data)
+    req = app_client.post("http://localhost:8000/v1/projects/novocab/learn", json=data)
     assert req.status_code == 503
diff --git a/tests/test_transform.py b/tests/test_transform.py
index 410f5e373..ad14f5489 100644
--- a/tests/test_transform.py
+++ b/tests/test_transform.py
@@ -7,8 +7,8 @@
 
 
 def test_parse_specs():
-    parsed = parse_specs('foo, bar(42,43,key=abc)')
-    assert parsed == [('foo', [], {}), ('bar', ['42', '43'], {'key': 'abc'})]
+    parsed = parse_specs("foo, bar(42,43,key=abc)")
+    assert parsed == [("foo", [], {}), ("bar", ["42", "43"], {"key": "abc"})]
 
 
 def test_get_transform_nonexistent():
@@ -32,8 +32,7 @@ def test_input_limiter_with_negative_value(project):
 
 
 def test_chained_transforms_text():
-    transf = annif.transform.get_transform(
-        "limit(5),pass,limit(3),", project=None)
+    transf = annif.transform.get_transform("limit(5),pass,limit(3),", project=None)
     assert transf.transform_text("abcdefghij") == "abc"
 
     # Check with a more arbitrary transform function
@@ -44,11 +43,9 @@ def test_chained_transforms_text():
 
 
 def test_chained_transforms_corpus(document_corpus):
-    transf = annif.transform.get_transform(
-        "limit(5),pass,limit(3),", project=None)
+    transf = annif.transform.get_transform("limit(5),pass,limit(3),", project=None)
     transformed_corpus = transf.transform_corpus(document_corpus)
-    for transf_doc, doc in zip(transformed_corpus.documents,
-                               document_corpus.documents):
+    for transf_doc, doc in zip(transformed_corpus.documents, document_corpus.documents):
         assert transf_doc.text == doc.text[:3]
         assert transf_doc.subject_set == doc.subject_set
 
@@ -56,7 +53,6 @@ def test_chained_transforms_corpus(document_corpus):
     reverser = annif.transform.transform.IdentityTransform(None)
     reverser.transform_fn = lambda x: x[::-1]
     transf.transforms.append(reverser)
-    for transf_doc, doc in zip(transformed_corpus.documents,
-                               document_corpus.documents):
+    for transf_doc, doc in zip(transformed_corpus.documents, document_corpus.documents):
         assert transf_doc.text == doc.text[:3][::-1]
         assert transf_doc.subject_set == doc.subject_set
diff --git a/tests/test_transform_langfilter.py b/tests/test_transform_langfilter.py
index b3a2f2d90..e4344eb28 100644
--- a/tests/test_transform_langfilter.py
+++ b/tests/test_transform_langfilter.py
@@ -27,7 +27,7 @@ def test_lang_filter(project):
         yhteiskunnalle. Kehitämme palveluja yhteistyössä kirjastojen,
         arkistojen, museoiden ja muiden toimijoiden kanssa.
     """
-    text = ' '.join(text.split())
+    text = " ".join(text.split())
     text_filtered = """
         Kansalliskirjasto on kaikille avoin kulttuuriperintöorganisaatio, joka
         palvelee valtakunnallisesti kansalaisia, tiedeyhteisöjä ja muita
@@ -39,7 +39,7 @@ def test_lang_filter(project):
         yhteiskunnalle. Kehitämme palveluja yhteistyössä kirjastojen,
         arkistojen, museoiden ja muiden toimijoiden kanssa.
     """
-    text_filtered = ' '.join(text_filtered.split())
+    text_filtered = " ".join(text_filtered.split())
     assert transf.transform_text(text) == text_filtered
 
 
@@ -48,17 +48,16 @@ def test_lang_filter_text_min_length(project):
     transf = annif.transform.get_transform("filter_lang", project)
     assert transf.transform_text(text) == text
     # Set a short text_min_length to apply language filtering:
-    transf = annif.transform.get_transform(
-        "filter_lang(text_min_length=50)", project)
+    transf = annif.transform.get_transform("filter_lang(text_min_length=50)", project)
     assert transf.transform_text(text) == ""
 
 
 def test_lang_filter_sentence_min_length(project):
     text = "This is a non-Finnish sentence of 42 chars. And this of 20 chars."
-    transf = annif.transform.get_transform(
-        "filter_lang(text_min_length=50)", project)
+    transf = annif.transform.get_transform("filter_lang(text_min_length=50)", project)
     assert transf.transform_text(text) == text
     # Set a short sentence_min_length to apply language filtering:
     transf = annif.transform.get_transform(
-        "filter_lang(text_min_length=50,sentence_min_length=30)", project)
+        "filter_lang(text_min_length=50,sentence_min_length=30)", project
+    )
     assert transf.transform_text(text) == "And this of 20 chars."
diff --git a/tests/test_util.py b/tests/test_util.py
index 0c1b2fa05..183333117 100644
--- a/tests/test_util.py
+++ b/tests/test_util.py
@@ -4,17 +4,7 @@
 
 
 def test_boolean():
-    inputs = [
-        '1',
-        '0',
-        'true',
-        'false',
-        'TRUE',
-        'FALSE',
-        'Yes',
-        'No',
-        True,
-        False]
+    inputs = ["1", "0", "true", "false", "TRUE", "FALSE", "Yes", "No", True, False]
     outputs = [True, False, True, False, True, False, True, False, True, False]
 
     for input, output in zip(inputs, outputs):
@@ -22,14 +12,8 @@ def test_boolean():
 
 
 def test_metric_code():
-    inputs = [
-        "F1 score (doc avg)",
-        "NDCG@10",
-        "True positives"]
-    outputs = [
-        "F1_score_doc_avg",
-        "NDCG@10",
-        "True_positives"]
+    inputs = ["F1 score (doc avg)", "NDCG@10", "True positives"]
+    outputs = ["F1_score_doc_avg", "NDCG@10", "True_positives"]
 
     for input, output in zip(inputs, outputs):
         assert annif.util.metric_code(input) == output
diff --git a/tests/test_vocab.py b/tests/test_vocab.py
index 3721b3bdc..6a4aa9921 100644
--- a/tests/test_vocab.py
+++ b/tests/test_vocab.py
@@ -9,64 +9,58 @@
 
 
 def load_dummy_vocab(tmpdir):
-    vocab = annif.vocab.AnnifVocabulary('vocab-id', str(tmpdir))
-    subjfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'dummy-subjects.tsv')
-    subjects = annif.corpus.SubjectFileTSV(subjfile, 'en')
+    vocab = annif.vocab.AnnifVocabulary("vocab-id", str(tmpdir))
+    subjfile = os.path.join(os.path.dirname(__file__), "corpora", "dummy-subjects.tsv")
+    subjects = annif.corpus.SubjectFileTSV(subjfile, "en")
     vocab.load_vocabulary(subjects)
     return vocab
 
 
 def test_get_vocab_invalid(registry):
     with pytest.raises(ValueError) as excinfo:
-        registry.get_vocab('', None)
-    assert 'Invalid vocabulary specification' in str(excinfo.value)
+        registry.get_vocab("", None)
+    assert "Invalid vocabulary specification" in str(excinfo.value)
 
 
 def test_get_vocab_hyphen(registry):
-    vocab, lang = registry.get_vocab('dummy-noname', None)
-    assert vocab.vocab_id == 'dummy-noname'
+    vocab, lang = registry.get_vocab("dummy-noname", None)
+    assert vocab.vocab_id == "dummy-noname"
     assert vocab is not None
 
 
 def test_update_subject_index_with_no_changes(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'dummy-subjects.tsv')
-    subjects = annif.corpus.SubjectFileTSV(subjfile, 'en')
+    subjfile = os.path.join(os.path.dirname(__file__), "corpora", "dummy-subjects.tsv")
+    subjects = annif.corpus.SubjectFileTSV(subjfile, "en")
 
     vocab.load_vocabulary(subjects)
     assert len(vocab.subjects) == 2
-    assert vocab.subjects.by_uri('http://example.org/dummy') == 0
-    assert vocab.subjects[0].uri == 'http://example.org/dummy'
-    assert vocab.subjects[0].labels['en'] == 'dummy'
+    assert vocab.subjects.by_uri("http://example.org/dummy") == 0
+    assert vocab.subjects[0].uri == "http://example.org/dummy"
+    assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri('http://example.org/none') == 1
-    assert vocab.subjects[1].uri == 'http://example.org/none'
-    assert vocab.subjects[1].labels['en'] == 'none'
-    assert vocab.subjects[1].notation == '42.42'
+    assert vocab.subjects.by_uri("http://example.org/none") == 1
+    assert vocab.subjects[1].uri == "http://example.org/none"
+    assert vocab.subjects[1].labels["en"] == "none"
+    assert vocab.subjects[1].notation == "42.42"
 
 
 def test_update_subject_index_with_removed_subject(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile_new = tmpdir.join('subjects_new.tsv')
+    subjfile_new = tmpdir.join("subjects_new.tsv")
     subjfile_new.write("<http://example.org/dummy>\tdummy\n")
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), 'en')
+    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
 
     vocab.load_vocabulary(subjects_new)
     assert len(vocab.subjects) == 2
-    assert vocab.subjects.by_uri('http://example.org/dummy') == 0
-    assert vocab.subjects[0].uri == 'http://example.org/dummy'
-    assert vocab.subjects[0].labels['en'] == 'dummy'
+    assert vocab.subjects.by_uri("http://example.org/dummy") == 0
+    assert vocab.subjects[0].uri == "http://example.org/dummy"
+    assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri('http://example.org/none') == 1
-    assert vocab.subjects[1].uri == 'http://example.org/none'
+    assert vocab.subjects.by_uri("http://example.org/none") == 1
+    assert vocab.subjects[1].uri == "http://example.org/none"
     assert vocab.subjects[1].labels is None
     assert vocab.subjects[1].notation is None
 
@@ -74,89 +68,95 @@ def test_update_subject_index_with_removed_subject(tmpdir):
 def test_update_subject_index_with_renamed_label_and_added_notation(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile_new = tmpdir.join('subjects_new.tsv')
-    subjfile_new.write("<http://example.org/dummy>\tdummy\n" +
-                       "<http://example.org/none>\tnew none\t42.42\n")
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), 'en')
+    subjfile_new = tmpdir.join("subjects_new.tsv")
+    subjfile_new.write(
+        "<http://example.org/dummy>\tdummy\n"
+        + "<http://example.org/none>\tnew none\t42.42\n"
+    )
+    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
 
     vocab.load_vocabulary(subjects_new)
     assert len(vocab.subjects) == 2
-    assert vocab.subjects.by_uri('http://example.org/dummy') == 0
-    assert vocab.subjects[0].uri == 'http://example.org/dummy'
-    assert vocab.subjects[0].labels['en'] == 'dummy'
+    assert vocab.subjects.by_uri("http://example.org/dummy") == 0
+    assert vocab.subjects[0].uri == "http://example.org/dummy"
+    assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri('http://example.org/none') == 1
-    assert vocab.subjects[1].uri == 'http://example.org/none'
-    assert vocab.subjects[1].labels['en'] == 'new none'
-    assert vocab.subjects[1].notation == '42.42'
+    assert vocab.subjects.by_uri("http://example.org/none") == 1
+    assert vocab.subjects[1].uri == "http://example.org/none"
+    assert vocab.subjects[1].labels["en"] == "new none"
+    assert vocab.subjects[1].notation == "42.42"
 
 
 def test_update_subject_index_with_added_subjects(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    subjfile_new = tmpdir.join('subjects_new.tsv')
-    subjfile_new.write("<http://example.org/dummy>\tdummy\n" +
-                       "<http://example.org/none>\tnone\n" +
-                       "<http://example.org/new-dummy>\tnew dummy\t42.42\n" +
-                       "<http://example.org/new-none>\tnew none\n")
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), 'en')
+    subjfile_new = tmpdir.join("subjects_new.tsv")
+    subjfile_new.write(
+        "<http://example.org/dummy>\tdummy\n"
+        + "<http://example.org/none>\tnone\n"
+        + "<http://example.org/new-dummy>\tnew dummy\t42.42\n"
+        + "<http://example.org/new-none>\tnew none\n"
+    )
+    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
 
     vocab.load_vocabulary(subjects_new)
     assert len(vocab.subjects) == 4
-    assert vocab.subjects.by_uri('http://example.org/dummy') == 0
-    assert vocab.subjects[0].uri == 'http://example.org/dummy'
-    assert vocab.subjects[0].labels['en'] == 'dummy'
+    assert vocab.subjects.by_uri("http://example.org/dummy") == 0
+    assert vocab.subjects[0].uri == "http://example.org/dummy"
+    assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri('http://example.org/new-dummy') == 2
-    assert vocab.subjects[2].uri == 'http://example.org/new-dummy'
-    assert vocab.subjects[2].labels['en'] == 'new dummy'
-    assert vocab.subjects[2].notation == '42.42'
+    assert vocab.subjects.by_uri("http://example.org/new-dummy") == 2
+    assert vocab.subjects[2].uri == "http://example.org/new-dummy"
+    assert vocab.subjects[2].labels["en"] == "new dummy"
+    assert vocab.subjects[2].notation == "42.42"
 
 
 def test_update_subject_index_force(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    subjfile_new = tmpdir.join('subjects_new.tsv')
-    subjfile_new.write("<http://example.org/dummy>\tdummy\n" +
-                       "<http://example.org/new-dummy>\tnew dummy\t42.42\n" +
-                       "<http://example.org/new-none>\tnew none\n")
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), 'en')
+    subjfile_new = tmpdir.join("subjects_new.tsv")
+    subjfile_new.write(
+        "<http://example.org/dummy>\tdummy\n"
+        + "<http://example.org/new-dummy>\tnew dummy\t42.42\n"
+        + "<http://example.org/new-none>\tnew none\n"
+    )
+    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
 
     vocab.load_vocabulary(subjects_new, force=True)
     assert len(vocab.subjects) == 3
-    assert vocab.subjects.by_uri('http://example.org/dummy') == 0
-    assert vocab.subjects[0].uri == 'http://example.org/dummy'
-    assert vocab.subjects[0].labels['en'] == 'dummy'
+    assert vocab.subjects.by_uri("http://example.org/dummy") == 0
+    assert vocab.subjects[0].uri == "http://example.org/dummy"
+    assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri('http://example.org/new-dummy') == 1
-    assert vocab.subjects[1].uri == 'http://example.org/new-dummy'
-    assert vocab.subjects[1].labels['en'] == 'new dummy'
-    assert vocab.subjects[1].notation == '42.42'
+    assert vocab.subjects.by_uri("http://example.org/new-dummy") == 1
+    assert vocab.subjects[1].uri == "http://example.org/new-dummy"
+    assert vocab.subjects[1].labels["en"] == "new dummy"
+    assert vocab.subjects[1].notation == "42.42"
 
 
 def test_skos(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    assert tmpdir.join('vocabs/vocab-id/subjects.ttl').exists()
-    assert tmpdir.join('vocabs/vocab-id/subjects.dump.gz').exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.ttl").exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
     assert isinstance(vocab.skos, annif.corpus.SubjectFileSKOS)
 
 
 def test_skos_cache(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    assert tmpdir.join('vocabs/vocab-id/subjects.ttl').exists()
-    assert tmpdir.join('vocabs/vocab-id/subjects.dump.gz').exists()
-    tmpdir.join('vocabs/vocab-id/subjects.dump.gz').remove()
-    assert not tmpdir.join('vocabs/vocab-id/subjects.dump.gz').exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.ttl").exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
+    tmpdir.join("vocabs/vocab-id/subjects.dump.gz").remove()
+    assert not tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
 
     assert isinstance(vocab.skos, annif.corpus.SubjectFileSKOS)
     # cached dump file has been recreated in .skos property access
-    assert tmpdir.join('vocabs/vocab-id/subjects.dump.gz').exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
 
 
 def test_skos_not_found(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    assert tmpdir.join('vocabs/vocab-id/subjects.ttl').exists()
-    assert tmpdir.join('vocabs/vocab-id/subjects.dump.gz').exists()
-    tmpdir.join('vocabs/vocab-id/subjects.ttl').remove()
-    tmpdir.join('vocabs/vocab-id/subjects.dump.gz').remove()
+    assert tmpdir.join("vocabs/vocab-id/subjects.ttl").exists()
+    assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
+    tmpdir.join("vocabs/vocab-id/subjects.ttl").remove()
+    tmpdir.join("vocabs/vocab-id/subjects.dump.gz").remove()
 
     with pytest.raises(NotInitializedException):
         vocab.skos
@@ -166,20 +166,15 @@ def test_as_graph(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
     graph = vocab.as_graph()
     labels = [
-        (str(tpl[0]), str(tpl[1]))
-        for tpl
-        in graph[
-            :rdflib.namespace.SKOS.prefLabel:]
+        (str(tpl[0]), str(tpl[1])) for tpl in graph[: rdflib.namespace.SKOS.prefLabel :]
     ]
     assert len(labels) == 2
-    assert ('http://example.org/dummy',	'dummy') in labels
-    assert ('http://example.org/none',	'none') in labels
+    assert ("http://example.org/dummy", "dummy") in labels
+    assert ("http://example.org/none", "none") in labels
     concepts = [
         str(tpl)
-        for tpl
-        in graph[
-            :rdflib.namespace.RDF.type:rdflib.namespace.SKOS.Concept]
+        for tpl in graph[: rdflib.namespace.RDF.type : rdflib.namespace.SKOS.Concept]
     ]
     assert len(concepts) == 2
-    assert 'http://example.org/dummy' in concepts
-    assert 'http://example.org/none' in concepts
+    assert "http://example.org/dummy" in concepts
+    assert "http://example.org/none" in concepts
diff --git a/tests/test_vocab_csv.py b/tests/test_vocab_csv.py
index fb8076052..0bc3820dd 100644
--- a/tests/test_vocab_csv.py
+++ b/tests/test_vocab_csv.py
@@ -5,78 +5,86 @@
 
 
 def test_recognize_csv_lowercase():
-    assert SubjectFileCSV.is_csv_file('subjects.csv')
+    assert SubjectFileCSV.is_csv_file("subjects.csv")
 
 
 def test_recognize_csv_uppercase():
-    assert SubjectFileCSV.is_csv_file('SUBJECTS.CSV')
+    assert SubjectFileCSV.is_csv_file("SUBJECTS.CSV")
 
 
 def test_recognize_tsv():
-    assert not SubjectFileCSV.is_csv_file('subjects.tsv')
+    assert not SubjectFileCSV.is_csv_file("subjects.tsv")
 
 
 def test_recognize_noext():
-    assert not SubjectFileCSV.is_csv_file('subjects')
+    assert not SubjectFileCSV.is_csv_file("subjects")
 
 
 def test_load_csv_uri_brackets(tmpdir):
-    tmpfile = tmpdir.join('subjects.csv')
-    tmpfile.write("uri,label_fi\n" +
-                  "<http://www.yso.fi/onto/yso/p8993>,hylyt\n" +
-                  "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi")
+    tmpfile = tmpdir.join("subjects.csv")
+    tmpfile.write(
+        "uri,label_fi\n"
+        + "<http://www.yso.fi/onto/yso/p8993>,hylyt\n"
+        + "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi"
+    )
 
     corpus = SubjectFileCSV(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
     assert subjects[0].notation is None
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
     assert subjects[1].notation is None
 
 
 def test_load_tsv_uri_nobrackets(tmpdir):
-    tmpfile = tmpdir.join('subjects.csv')
-    tmpfile.write("uri,label_fi\n" +
-                  "http://www.yso.fi/onto/yso/p8993,hylyt\n" +
-                  "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi")
+    tmpfile = tmpdir.join("subjects.csv")
+    tmpfile.write(
+        "uri,label_fi\n"
+        + "http://www.yso.fi/onto/yso/p8993,hylyt\n"
+        + "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi"
+    )
 
     corpus = SubjectFileCSV(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
     assert subjects[0].notation is None
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
     assert subjects[1].notation is None
 
 
 def test_load_tsv_with_notations(tmpdir):
-    tmpfile = tmpdir.join('subjects-with-notations.csv')
-    tmpfile.write("uri,label_fi,notation\n" +
-                  "http://www.yso.fi/onto/yso/p8993,hylyt,42.42\n" +
-                  "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi,42.0")
+    tmpfile = tmpdir.join("subjects-with-notations.csv")
+    tmpfile.write(
+        "uri,label_fi,notation\n"
+        + "http://www.yso.fi/onto/yso/p8993,hylyt,42.42\n"
+        + "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi,42.0"
+    )
 
     corpus = SubjectFileCSV(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
-    assert subjects[0].notation == '42.42'
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
-    assert subjects[1].notation == '42.0'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
+    assert subjects[0].notation == "42.42"
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
+    assert subjects[1].notation == "42.0"
 
 
 def test_load_tsv_with_deprecated(tmpdir):
-    tmpfile = tmpdir.join('subjects.csv')
-    tmpfile.write("uri,label_fi\n" +
-                  "<http://www.yso.fi/onto/yso/p8993>,hylyt\n" +
-                  "<http://example.org/deprecated>,\n" +
-                  "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi")
+    tmpfile = tmpdir.join("subjects.csv")
+    tmpfile.write(
+        "uri,label_fi\n"
+        + "<http://www.yso.fi/onto/yso/p8993>,hylyt\n"
+        + "<http://example.org/deprecated>,\n"
+        + "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi"
+    )
 
     corpus = SubjectFileCSV(str(tmpfile))
     subjects = list(corpus.subjects)
@@ -88,9 +96,9 @@ def test_load_tsv_with_deprecated(tmpdir):
 
     active = list(index.active)
     assert len(active) == 2
-    assert active[0][1].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert active[0][1].labels['fi'] == 'hylyt'
+    assert active[0][1].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert active[0][1].labels["fi"] == "hylyt"
     assert active[0][1].notation is None
-    assert active[1][1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert active[1][1].labels['fi'] == 'neoliittinen kausi'
+    assert active[1][1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert active[1][1].labels["fi"] == "neoliittinen kausi"
     assert active[1][1].notation is None
diff --git a/tests/test_vocab_skos.py b/tests/test_vocab_skos.py
index 49359eb86..a3c1e508c 100644
--- a/tests/test_vocab_skos.py
+++ b/tests/test_vocab_skos.py
@@ -7,28 +7,29 @@
 
 
 def test_recognize_turtle():
-    assert SubjectFileSKOS.is_rdf_file('subjects.ttl')
+    assert SubjectFileSKOS.is_rdf_file("subjects.ttl")
 
 
 def test_recognize_rdfxml():
-    assert SubjectFileSKOS.is_rdf_file('subjects.rdf')
+    assert SubjectFileSKOS.is_rdf_file("subjects.rdf")
 
 
 def test_recognize_nt():
-    assert SubjectFileSKOS.is_rdf_file('subjects.nt')
+    assert SubjectFileSKOS.is_rdf_file("subjects.nt")
 
 
 def test_recognize_tsv():
-    assert not SubjectFileSKOS.is_rdf_file('subjects.tsv')
+    assert not SubjectFileSKOS.is_rdf_file("subjects.tsv")
 
 
 def test_recognize_noext():
-    assert not SubjectFileSKOS.is_rdf_file('subjects')
+    assert not SubjectFileSKOS.is_rdf_file("subjects")
 
 
 def test_load_turtle(tmpdir):
-    tmpfile = tmpdir.join('subjects.ttl')
-    tmpfile.write("""
+    tmpfile = tmpdir.join("subjects.ttl")
+    tmpfile.write(
+        """
 @prefix skos: <http://www.w3.org/2004/02/skos/core#> .
 @prefix yso: <http://www.yso.fi/onto/yso/> .
 @prefix owl: <http://www.w3.org/2002/07/owl#>.
@@ -44,19 +45,21 @@ def test_load_turtle(tmpdir):
     skos:broader yso:p4624 ;
     skos:prefLabel "Neolithic period"@en, "neoliittinen kausi"@fi,
         "neolitisk tid"@sv .
-    """)
+    """
+    )
 
     corpus = SubjectFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 1  # one of the concepts was deprecated
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
     assert subjects[0].notation is None
 
 
 def test_load_turtle_with_notation(tmpdir):
-    tmpfile = tmpdir.join('subjects.ttl')
-    tmpfile.write("""
+    tmpfile = tmpdir.join("subjects.ttl")
+    tmpfile.write(
+        """
 @prefix skos: <http://www.w3.org/2004/02/skos/core#> .
 @prefix yso: <http://www.yso.fi/onto/yso/> .
 @prefix owl: <http://www.w3.org/2002/07/owl#>.
@@ -66,18 +69,20 @@ def test_load_turtle_with_notation(tmpdir):
     skos:prefLabel "hylyt"@fi, "shipwrecks (objects)"@en, "vrak"@sv ;
     skos:notation "42.42" ;
     skos:related yso:p8869 .
-    """)
+    """
+    )
 
     corpus = SubjectFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
-    assert subjects[0].notation == '42.42'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
+    assert subjects[0].notation == "42.42"
 
 
 def test_load_turtle_missing_langtags(tmpdir):
-    tmpfile = tmpdir.join('subjects.ttl')
-    tmpfile.write("""
+    tmpfile = tmpdir.join("subjects.ttl")
+    tmpfile.write(
+        """
 @prefix skos: <http://www.w3.org/2004/02/skos/core#> .
 @prefix ex: <http://example.org/> .
 
@@ -91,28 +96,27 @@ def test_load_turtle_missing_langtags(tmpdir):
 
 # Concept without a prefLabel
 ex:conc3 a skos:Concept .
-    """)
+    """
+    )
 
     corpus = SubjectFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 3
 
     # check that the vocabulary contains the expected labels
-    en_labels = {subj.labels['en'] for subj in subjects}
-    assert 'Concept 1' in en_labels
-    assert 'Concept 2' in en_labels
-    assert 'ex:conc3' in en_labels
+    en_labels = {subj.labels["en"] for subj in subjects}
+    assert "Concept 1" in en_labels
+    assert "Concept 2" in en_labels
+    assert "ex:conc3" in en_labels
 
 
 def test_load_turtle_get_languages(testdatadir):
     subjectfile = os.path.join(
-        os.path.dirname(__file__),
-        'corpora',
-        'archaeology',
-        'yso-archaeology.ttl')
+        os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
+    )
     corpus = SubjectFileSKOS(subjectfile)
     langs = corpus.languages
     assert len(langs) == 3
-    assert 'fi' in langs
-    assert 'sv' in langs
-    assert 'en' in langs
+    assert "fi" in langs
+    assert "sv" in langs
+    assert "en" in langs
diff --git a/tests/test_vocab_tsv.py b/tests/test_vocab_tsv.py
index 5ff90a697..1fc318e0f 100644
--- a/tests/test_vocab_tsv.py
+++ b/tests/test_vocab_tsv.py
@@ -5,60 +5,68 @@
 
 
 def test_load_tsv_uri_brackets(tmpdir):
-    tmpfile = tmpdir.join('subjects.tsv')
-    tmpfile.write("<http://www.yso.fi/onto/yso/p8993>\thylyt\n" +
-                  "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi")
+    tmpfile = tmpdir.join("subjects.tsv")
+    tmpfile.write(
+        "<http://www.yso.fi/onto/yso/p8993>\thylyt\n"
+        + "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi"
+    )
 
-    corpus = SubjectFileTSV(str(tmpfile), 'fi')
+    corpus = SubjectFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
     assert subjects[0].notation is None
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
     assert subjects[1].notation is None
 
 
 def test_load_tsv_uri_nobrackets(tmpdir):
-    tmpfile = tmpdir.join('subjects.tsv')
-    tmpfile.write("http://www.yso.fi/onto/yso/p8993\thylyt\n" +
-                  "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi")
+    tmpfile = tmpdir.join("subjects.tsv")
+    tmpfile.write(
+        "http://www.yso.fi/onto/yso/p8993\thylyt\n"
+        + "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi"
+    )
 
-    corpus = SubjectFileTSV(str(tmpfile), 'fi')
+    corpus = SubjectFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
     assert subjects[0].notation is None
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
     assert subjects[1].notation is None
 
 
 def test_load_tsv_with_notations(tmpdir):
-    tmpfile = tmpdir.join('subjects-with-notations.tsv')
-    tmpfile.write("http://www.yso.fi/onto/yso/p8993\thylyt\t42.42\n" +
-                  "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi\t42.0")
+    tmpfile = tmpdir.join("subjects-with-notations.tsv")
+    tmpfile.write(
+        "http://www.yso.fi/onto/yso/p8993\thylyt\t42.42\n"
+        + "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi\t42.0"
+    )
 
-    corpus = SubjectFileTSV(str(tmpfile), 'fi')
+    corpus = SubjectFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
-    assert subjects[0].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert subjects[0].labels['fi'] == 'hylyt'
-    assert subjects[0].notation == '42.42'
-    assert subjects[1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert subjects[1].labels['fi'] == 'neoliittinen kausi'
-    assert subjects[1].notation == '42.0'
+    assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert subjects[0].labels["fi"] == "hylyt"
+    assert subjects[0].notation == "42.42"
+    assert subjects[1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert subjects[1].labels["fi"] == "neoliittinen kausi"
+    assert subjects[1].notation == "42.0"
 
 
 def test_load_tsv_with_deprecated(tmpdir):
-    tmpfile = tmpdir.join('subjects.tsv')
-    tmpfile.write("<http://www.yso.fi/onto/yso/p8993>\thylyt\n" +
-                  "<http://example.org/deprecated>\t\n" +
-                  "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi")
-
-    corpus = SubjectFileTSV(str(tmpfile), 'fi')
+    tmpfile = tmpdir.join("subjects.tsv")
+    tmpfile.write(
+        "<http://www.yso.fi/onto/yso/p8993>\thylyt\n"
+        + "<http://example.org/deprecated>\t\n"
+        + "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi"
+    )
+
+    corpus = SubjectFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(list(corpus.subjects)) == 3
     assert subjects[1].labels is None
@@ -68,9 +76,9 @@ def test_load_tsv_with_deprecated(tmpdir):
 
     active = list(index.active)
     assert len(active) == 2
-    assert active[0][1].uri == 'http://www.yso.fi/onto/yso/p8993'
-    assert active[0][1].labels['fi'] == 'hylyt'
+    assert active[0][1].uri == "http://www.yso.fi/onto/yso/p8993"
+    assert active[0][1].labels["fi"] == "hylyt"
     assert active[0][1].notation is None
-    assert active[1][1].uri == 'http://www.yso.fi/onto/yso/p9285'
-    assert active[1][1].labels['fi'] == 'neoliittinen kausi'
+    assert active[1][1].uri == "http://www.yso.fi/onto/yso/p9285"
+    assert active[1][1].labels["fi"] == "neoliittinen kausi"
     assert active[1][1].notation is None