Repository: NatLibFi/Annif
Pull request #840: Subject exclusion support
Author: login: osma
Labels: name: enhancement
Created: 2025-03-26T08:19:54Z | Merged: 2025-03-28T15:41:41Z

Fixes #735

This PR makes it possible to exclude specific subjects from a vocabulary by their URIs. It is controlled by a keyword parameter for the `vocab` setting, like this:

    vocab=yso(en,exclude=http://www.yso.fi/onto/yso/p12345)

or for the ZBW / STW Thesaurus case (excluding the frequent false positives Theory and USA - note that `|` is used as the separator between URIs):

    vocab=stw(en,exclude=http://zbw.eu/stw/descriptor/19073-6|http://zbw.eu/stw/descriptor/17829-1)

# Refactoring

This turned into quite a refactoring exercise so the PR is quite large with lots of churn, even though the filtering/exclusion functionality itself isn't that much extra code. 

I had to adjust the responsibilities of various classes (AnnifProject, AnnifRegistry, AnnifVocabulary, SubjectIndex) to make it possible to implement SubjectIndexFilter. I also abstracted SubjectIndex into an abstract base class and two concrete implementations, SubjectIndexFile (basically what was in the old SubjectIndex) and SubjectIndexFilter (new class that implements the exclusion).

The old module structure related to the vocabulary and subject index functionality was quite messy, so I ended up separating vocabulary-specific functionality into `annif.vocab` (now a directory) and moved several classes into this new module from `annif.corpus` where they had been located mixed up with classes related to document corpora.

I also renamed some classes: `SubjectCorpus` -> `VocabSource` and `SubjectFile*` -> `VocabFile*`. I think these names better represent their function.

# Quality assurance complaints

SonarCloud has 1 complaint and CodeClimate shows 8 complaint. These were all in code that was moved around in this PR, but the issues were not introduced in this PR.

# Future work

This was just the first step in supporting exclusion. I would like to continue with more features after this PR:

- support different kinds of include/exclude rules, e.g. by concept type, concept scheme and collection - see https://github.com/NatLibFi/Annif/issues/538#issuecomment-979109206
- support simple wildcards in the exclude rules, for example to exclude specific URI namespaces, or a general `exclude=*` rule(followed by include rules) - this could probably be implemented using the [fnmatch](https://docs.python.org/3/library/fnmatch.html) standard library
- dynamic exclude rules at request time - see https://github.com/NatLibFi/Annif/issues/735#issuecomment-2066415020

Linked issue #735: Dealing with overrepresented concepts / blacklisting
Several institutions have observed that some models / ensembles struggle with concepts that are overrepresented in the training data so that they are suggested way to often. One fix for that is to identify rules that limit the contexts in which those concepts can be suggested. Could we implement something in Annif that allows specifying those rules?

CC @schlawiner @lakshmi-bashyam

related: https://github.com/NatLibFi/Annif/issues/538 ; https://github.com/NatLibFi/Annif/issues/596
Linked issue comments: user: login: osma
created_at: 2023-09-13T18:41:55Z
body: Thank you for the suggestion. Indeed this seems like a recurring problem, so a generic mechanism could be useful.

This was one of the ideas discussed in issue #538, especially in https://github.com/NatLibFi/Annif/issues/538#issuecomment-979109206 . But there were maybe too many ideas thrown around and so far nothing has been implemented. So let's keep this issue focused on only the problem of overrepresented concepts and the possible solution to make it possible to block problematic concepts, since it seems that both ZBW and ZPID have already decided to use such a mechanism implemented outside Annif.

I think this configuration example from https://github.com/NatLibFi/Annif/issues/538#issuecomment-976532224 is still valid:

```
[omikuji_stw_en]
vocab=stw_9_10
exclude_concepts=http://zbw.eu/stw/descriptor/19073-6,http://zbw.eu/stw/descriptor/17829-1
backend=omikuji
```

and the meaning of this would be that the two concepts (USA and Theory) listed in `exclude_concepts` are ignored both when reading/processing training data and when generating suggestions, but only for this particular project. There could still be other projects using the same vocabulary and the setting would of course not affect those. So in an ensemble, it would be possible to block specific concepts on the level of a particular backend project, if it has a tendency to suggest certain concepts too often without good reason.

As noted in #538, it would make sense to avoid the term "blacklisting" due to [connotations](https://www.splunk.com/en_us/blog/learn/blacklist-whitelist-inclusivity.html). I think "exclude", "block" or "deny" are all valid alternatives.
author_association: MEMBER
user: login: osma
created_at: 2023-09-22T13:40:29Z
body: I've thought about the best way to implement something like this in Annif code.

I think this should be a general mechanism and ideally no changes to individual backend implementations should be necessary. This means that the setting should be handled on the level of AnnifProject. One possibility is that SubjectIndex would be made aware of the blocked/excluded concepts, similar to how it already handles deprecated concepts.

For the configuration, this could be implemented as an extra option to the `vocab` setting. There is already a mechanism to set the vocabulary language using a setting such as `vocab=lcsh(en)`. We could extend that to take another parameter, like this:

    vocab=stw(en,exclude=http://zbw.eu/stw/descriptor/19073-6 http://zbw.eu/stw/descriptor/17829-1)

When there's no need to set the language, this could work as well:

    vocab=stw(exclude=http://zbw.eu/stw/descriptor/19073-6 http://zbw.eu/stw/descriptor/17829-1)

One minor syntax consideration here is that commas are already used to separate different parameters, so it's not possible to use commas as a separator between concept URIs. Above I've used spaces instead, but other symbols such as `|` (pipe) could work as well - as long as they are not used in URIs.
author_association: MEMBER
user: login: juhoinkinen
created_at: 2024-04-19T11:54:26Z
body: Just throwing in the idea: could the denylisting be (also) "dynamic", in the sense that the suggest request could include a parameter containing the concepts that are not wanted at that particular time? I think there could be some users of Annif API that could benefit from this.

This could be useful for e.g. university repositories, as very many theses and dissertations get the ["final projects (education)"](http://www.yso.fi/onto/yso/p1380) concept as a unwanted and redundant suggestion. I assume they now exclude that concept in their own system(?) to not show it to the student.

Another use case would be to restrict the suggestions using the ontology hierarchy e.g. to only all [physical objects](https://finto.fi/yso/en/page/p1435) or some groups. There could be even a UI component where a user could select the allowed or denied concepts in the hierarchy tree. That would be cool, but maybe not so useful.
author_association: MEMBER

Diff:
diff --git a/annif/cli.py b/annif/cli.py
index 1f0070732..ca20fd4c5 100644
--- a/annif/cli.py
+++ b/annif/cli.py
@@ -142,8 +142,8 @@ def run_list_vocabs():
 
 @cli.command("load-vocab")
 @click.argument("vocab_id", shell_complete=cli_util.complete_param)
-@click.argument("subjectfile", type=click.Path(exists=True, dir_okay=False))
-@click.option("--language", "-L", help="Language of subject file")
+@click.argument("vocab_file", type=click.Path(exists=True, dir_okay=False))
+@click.option("--language", "-L", help="Language of TSV vocabulary file")
 @click.option(
     "--force",
     "-f",
@@ -152,19 +152,19 @@ def run_list_vocabs():
     help="Replace existing vocabulary completely instead of updating it",
 )
 @cli_util.common_options
-def run_load_vocab(vocab_id, language, force, subjectfile):
+def run_load_vocab(vocab_id, language, force, vocab_file):
     """
     Load a vocabulary from a subject file.
     """
     vocab = cli_util.get_vocab(vocab_id)
-    if annif.corpus.SubjectFileSKOS.is_rdf_file(subjectfile):
+    if annif.vocab.VocabFileSKOS.is_rdf_file(vocab_file):
         # SKOS/RDF file supported by rdflib
-        subjects = annif.corpus.SubjectFileSKOS(subjectfile)
-        click.echo(f"Loading vocabulary from SKOS file {subjectfile}...")
-    elif annif.corpus.SubjectFileCSV.is_csv_file(subjectfile):
+        vocab_file = annif.vocab.VocabFileSKOS(vocab_file)
+        click.echo(f"Loading vocabulary from SKOS file {vocab_file}...")
+    elif annif.vocab.VocabFileCSV.is_csv_file(vocab_file):
         # CSV file
-        subjects = annif.corpus.SubjectFileCSV(subjectfile)
-        click.echo(f"Loading vocabulary from CSV file {subjectfile}...")
+        vocab_file = annif.vocab.VocabFileCSV(vocab_file)
+        click.echo(f"Loading vocabulary from CSV file {vocab_file}...")
     else:
         # probably a TSV file - we need to know its language
         if not language:
@@ -173,9 +173,9 @@ def run_load_vocab(vocab_id, language, force, subjectfile):
                 err=True,
             )
             sys.exit(1)
-        click.echo(f"Loading vocabulary from TSV file {subjectfile}...")
-        subjects = annif.corpus.SubjectFileTSV(subjectfile, language)
-    vocab.load_vocabulary(subjects, force=force)
+        click.echo(f"Loading vocabulary from TSV file {vocab_file}...")
+        vocab_file = annif.vocab.VocabFileTSV(vocab_file, language)
+    vocab.load_vocabulary(vocab_file, force=force)
 
 
 @cli.command("train")
diff --git a/annif/corpus/__init__.py b/annif/corpus/__init__.py
index fb0ceef86..9cb2df596 100644
--- a/annif/corpus/__init__.py
+++ b/annif/corpus/__init__.py
@@ -8,20 +8,13 @@
     LimitingDocumentCorpus,
     TransformingDocumentCorpus,
 )
-from .skos import SubjectFileSKOS
-from .subject import Subject, SubjectFileCSV, SubjectFileTSV, SubjectIndex, SubjectSet
-from .types import Document
+from .types import Document, SubjectSet
 
 __all__ = [
     "DocumentDirectory",
     "DocumentFile",
     "DocumentList",
-    "Subject",
-    "SubjectFileTSV",
-    "SubjectFileCSV",
-    "SubjectIndex",
     "SubjectSet",
-    "SubjectFileSKOS",
     "Document",
     "CombinedCorpus",
     "TransformingDocumentCorpus",
diff --git a/annif/corpus/document.py b/annif/corpus/document.py
index 8d4ef5319..d751dab53 100644
--- a/annif/corpus/document.py
+++ b/annif/corpus/document.py
@@ -11,8 +11,7 @@
 
 import annif.util
 
-from .subject import SubjectSet
-from .types import Document, DocumentCorpus
+from .types import Document, DocumentCorpus, SubjectSet
 
 if TYPE_CHECKING:
     from collections.abc import Iterator
diff --git a/annif/corpus/subject.py b/annif/corpus/subject.py
deleted file mode 100644
index 2e3ffe5de..000000000
--- a/annif/corpus/subject.py
+++ /dev/null
@@ -1,295 +0,0 @@
-"""Classes for supporting subject corpora expressed as directories or files"""
-
-from __future__ import annotations
-
-import csv
-import os.path
-from typing import TYPE_CHECKING, Any
-
-import annif
-import annif.util
-
-from .skos import serialize_subjects_to_skos
-from .types import Subject, SubjectCorpus
-
-if TYPE_CHECKING:
-    from collections.abc import Generator, Iterator
-
-    import numpy as np
-
-logger = annif.logger.getChild("subject")
-logger.addFilter(annif.util.DuplicateFilter())
-
-
-class SubjectFileTSV(SubjectCorpus):
-    """A monolingual subject vocabulary stored in a TSV file."""
-
-    def __init__(self, path: str, language: str) -> None:
-        """initialize the SubjectFileTSV given a path to a TSV file and the
-        language of the vocabulary"""
-
-        self.path = path
-        self.language = language
-
-    def _parse_line(self, line: str) -> Iterator[Subject]:
-        vals = line.strip().split("\t", 2)
-        clean_uri = annif.util.cleanup_uri(vals[0])
-        label = vals[1] if len(vals) >= 2 else None
-        labels = {self.language: label} if label else None
-        notation = vals[2] if len(vals) >= 3 else None
-        yield Subject(uri=clean_uri, labels=labels, notation=notation)
-
-    @property
-    def languages(self) -> list[str]:
-        return [self.language]
-
-    @property
-    def subjects(self) -> Generator:
-        with open(self.path, encoding="utf-8-sig") as subjfile:
-            for line in subjfile:
-                yield from self._parse_line(line)
-
-    def save_skos(self, path: str) -> None:
-        """Save the contents of the subject vocabulary into a SKOS/Turtle
-        file with the given path name."""
-        serialize_subjects_to_skos(self.subjects, path)
-
-
-class SubjectFileCSV(SubjectCorpus):
-    """A multilingual subject vocabulary stored in a CSV file."""
-
-    def __init__(self, path: str) -> None:
-        """initialize the SubjectFileCSV given a path to a CSV file"""
-        self.path = path
-
-    def _parse_row(self, row: dict[str, str]) -> Iterator[Subject]:
-        labels = {
-            fname.replace("label_", ""): value or None
-            for fname, value in row.items()
-            if fname.startswith("label_")
-        }
-
-        # if there are no labels in any language, set labels to None
-        # indicating a deprecated subject
-        if set(labels.values()) == {None}:
-            labels = None
-
-        yield Subject(
-            uri=annif.util.cleanup_uri(row["uri"]),
-            labels=labels,
-            notation=row.get("notation", None) or None,
-        )
-
-    @property
-    def languages(self) -> list[str]:
-        # infer the supported languages from the CSV column names
-        with open(self.path, encoding="utf-8-sig") as csvfile:
-            reader = csv.reader(csvfile)
-            fieldnames = next(reader, None)
-
-        return [
-            fname.replace("label_", "")
-            for fname in fieldnames
-            if fname.startswith("label_")
-        ]
-
-    @property
-    def subjects(self) -> Generator:
-        with open(self.path, encoding="utf-8-sig") as csvfile:
-            reader = csv.DictReader(csvfile)
-            for row in reader:
-                yield from self._parse_row(row)
-
-    def save_skos(self, path: str) -> None:
-        """Save the contents of the subject vocabulary into a SKOS/Turtle
-        file with the given path name."""
-        serialize_subjects_to_skos(self.subjects, path)
-
-    @staticmethod
-    def is_csv_file(path: str) -> bool:
-        """return True if the path looks like a CSV file"""
-
-        return os.path.splitext(path)[1].lower() == ".csv"
-
-
-class SubjectIndex:
-    """An index that remembers the associations between integers subject IDs
-    and their URIs and labels."""
-
-    def __init__(self) -> None:
-        self._subjects = []
-        self._uri_idx = {}
-        self._label_idx = {}
-        self._languages = None
-
-    def load_subjects(self, corpus: SubjectCorpus) -> None:
-        """Initialize the subject index from a subject corpus"""
-
-        self._languages = corpus.languages
-        for subject in corpus.subjects:
-            self.append(subject)
-
-    def __len__(self) -> int:
-        return len(self._subjects)
-
-    @property
-    def languages(self) -> list[str] | None:
-        return self._languages
-
-    def __getitem__(self, subject_id: int) -> Subject:
-        return self._subjects[subject_id]
-
-    def append(self, subject: Subject) -> None:
-        if self._languages is None and subject.labels is not None:
-            self._languages = list(subject.labels.keys())
-
-        subject_id = len(self._subjects)
-        self._uri_idx[subject.uri] = subject_id
-        if subject.labels:
-            for lang, label in subject.labels.items():
-                self._label_idx[(label, lang)] = subject_id
-        self._subjects.append(subject)
-
-    def contains_uri(self, uri: str) -> bool:
-        return uri in self._uri_idx
-
-    def by_uri(self, uri: str, warnings: bool = True) -> int | None:
-        """return the subject ID of a subject by its URI, or None if not found.
-        If warnings=True, log a warning message if the URI cannot be found."""
-        try:
-            return self._uri_idx[uri]
-        except KeyError:
-            if warnings:
-                logger.warning("Unknown subject URI <%s>", uri)
-            return None
-
-    def by_label(self, label: str | None, language: str) -> int | None:
-        """return the subject ID of a subject by its label in a given
-        language"""
-        try:
-            return self._label_idx[(label, language)]
-        except KeyError:
-            logger.warning('Unknown subject label "%s"@%s', label, language)
-            return None
-
-    def deprecated_ids(self) -> list[int]:
-        """return indices of deprecated subjects"""
-
-        return [
-            subject_id
-            for subject_id, subject in enumerate(self._subjects)
-            if subject.labels is None
-        ]
-
-    @property
-    def active(self) -> list[tuple[int, Subject]]:
-        """return a list of (subject_id, subject) tuples of all subjects that
-        are not deprecated"""
-
-        return [
-            (subj_id, subject)
-            for subj_id, subject in enumerate(self._subjects)
-            if subject.labels is not None
-        ]
-
-    def save(self, path: str) -> None:
-        """Save this subject index into a file with the given path name."""
-
-        fieldnames = ["uri", "notation"] + [f"label_{lang}" for lang in self._languages]
-
-        with open(path, "w", encoding="utf-8", newline="") as csvfile:
-            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
-            writer.writeheader()
-            for subject in self:
-                row = {"uri": subject.uri, "notation": subject.notation or ""}
-                if subject.labels:
-                    for lang, label in subject.labels.items():
-                        row[f"label_{lang}"] = label
-                writer.writerow(row)
-
-    @classmethod
-    def load(cls, path: str) -> SubjectIndex:
-        """Load a subject index from a CSV file and return it."""
-
-        corpus = SubjectFileCSV(path)
-        subject_index = cls()
-        subject_index.load_subjects(corpus)
-        return subject_index
-
-
-class SubjectSet:
-    """Represents a set of subjects for a document."""
-
-    def __init__(self, subject_ids: Any | None = None) -> None:
-        """Create a SubjectSet and optionally initialize it from an iterable
-        of subject IDs"""
-
-        if subject_ids:
-            # use set comprehension to eliminate possible duplicates
-            self._subject_ids = list(
-                {subject_id for subject_id in subject_ids if subject_id is not None}
-            )
-        else:
-            self._subject_ids = []
-
-    def __len__(self) -> int:
-        return len(self._subject_ids)
-
-    def __getitem__(self, idx: int) -> int:
-        return self._subject_ids[idx]
-
-    def __bool__(self) -> bool:
-        return bool(self._subject_ids)
-
-    def __eq__(self, other: Any) -> bool:
-        if isinstance(other, SubjectSet):
-            return self._subject_ids == other._subject_ids
-
-        return False
-
-    @classmethod
-    def from_string(
-        cls, subj_data: str, subject_index: SubjectIndex, language: str
-    ) -> SubjectSet:
-        subject_ids = set()
-        for line in subj_data.splitlines():
-            uri, label = cls._parse_line(line)
-            if uri is not None:
-                subject_ids.add(subject_index.by_uri(uri))
-            else:
-                subject_ids.add(subject_index.by_label(label, language))
-        return cls(subject_ids)
-
-    @staticmethod
-    def _parse_line(
-        line: str,
-    ) -> tuple[str | None, str | None]:
-        uri = label = None
-        vals = line.split("\t")
-        for val in vals:
-            val = val.strip()
-            if val == "":
-                continue
-            if val.startswith("<") and val.endswith(">"):  # URI
-                uri = val[1:-1]
-                continue
-            label = val
-            break
-        return uri, label
-
-    def as_vector(
-        self, size: int | None = None, destination: np.ndarray | None = None
-    ) -> np.ndarray:
-        """Return the hits as a one-dimensional NumPy array in sklearn
-        multilabel indicator format. Use destination array if given (not
-        None), otherwise create and return a new one of the given size."""
-
-        if destination is None:
-            import numpy as np
-
-            assert size is not None and size > 0
-            destination = np.zeros(size, dtype=bool)
-
-        destination[list(self._subject_ids)] = True
-
-        return destination
diff --git a/annif/corpus/types.py b/annif/corpus/types.py
index e94a8c1d9..a166f569b 100644
--- a/annif/corpus/types.py
+++ b/annif/corpus/types.py
@@ -1,10 +1,16 @@
-"""Basic types for document and subject corpora"""
+"""Basic types for document corpora"""
 
 from __future__ import annotations
 
 import abc
 import collections
 from itertools import islice
+from typing import TYPE_CHECKING, Any
+
+if TYPE_CHECKING:
+    import numpy as np
+
+    from annif.vocab import SubjectIndex
 
 Document = collections.namedtuple("Document", "text subject_set")
 
@@ -40,27 +46,79 @@ def is_empty(self) -> bool:
             return True
 
 
-Subject = collections.namedtuple("Subject", "uri labels notation")
-
-
-class SubjectCorpus(metaclass=abc.ABCMeta):
-    """Abstract base class for subject corpora"""
-
-    @property
-    @abc.abstractmethod
-    def subjects(self):
-        """Iterate through the subject corpus, yielding Subject objects."""
-        pass  # pragma: no cover
-
-    @property
-    @abc.abstractmethod
-    def languages(self):
-        """Provide a list of language codes supported by this subject
-        corpus."""
-        pass  # pragma: no cover
-
-    @abc.abstractmethod
-    def save_skos(self, path):
-        """Save the contents of the subject corpus into a SKOS/Turtle
-        file with the given path name."""
-        pass  # pragma: no cover
+class SubjectSet:
+    """Represents a set of subjects for a document."""
+
+    def __init__(self, subject_ids: Any | None = None) -> None:
+        """Create a SubjectSet and optionally initialize it from an iterable
+        of subject IDs"""
+
+        if subject_ids:
+            # use set comprehension to eliminate possible duplicates
+            self._subject_ids = list(
+                {subject_id for subject_id in subject_ids if subject_id is not None}
+            )
+        else:
+            self._subject_ids = []
+
+    def __len__(self) -> int:
+        return len(self._subject_ids)
+
+    def __getitem__(self, idx: int) -> int:
+        return self._subject_ids[idx]
+
+    def __bool__(self) -> bool:
+        return bool(self._subject_ids)
+
+    def __eq__(self, other: Any) -> bool:
+        if isinstance(other, SubjectSet):
+            return self._subject_ids == other._subject_ids
+
+        return False
+
+    @classmethod
+    def from_string(
+        cls, subj_data: str, subject_index: SubjectIndex, language: str
+    ) -> SubjectSet:
+        subject_ids = set()
+        for line in subj_data.splitlines():
+            uri, label = cls._parse_line(line)
+            if uri is not None:
+                subject_ids.add(subject_index.by_uri(uri))
+            else:
+                subject_ids.add(subject_index.by_label(label, language))
+        return cls(subject_ids)
+
+    @staticmethod
+    def _parse_line(
+        line: str,
+    ) -> tuple[str | None, str | None]:
+        uri = label = None
+        vals = line.split("\t")
+        for val in vals:
+            val = val.strip()
+            if val == "":
+                continue
+            if val.startswith("<") and val.endswith(">"):  # URI
+                uri = val[1:-1]
+                continue
+            label = val
+            break
+        return uri, label
+
+    def as_vector(
+        self, size: int | None = None, destination: np.ndarray | None = None
+    ) -> np.ndarray:
+        """Return the hits as a one-dimensional NumPy array in sklearn
+        multilabel indicator format. Use destination array if given (not
+        None), otherwise create and return a new one of the given size."""
+
+        if destination is None:
+            import numpy as np
+
+            assert size is not None and size > 0
+            destination = np.zeros(size, dtype=bool)
+
+        destination[list(self._subject_ids)] = True
+
+        return destination
diff --git a/annif/project.py b/annif/project.py
index 130e53b50..1b7134a88 100644
--- a/annif/project.py
+++ b/annif/project.py
@@ -4,6 +4,7 @@
 
 import enum
 import os.path
+import re
 from shutil import rmtree
 from typing import TYPE_CHECKING
 
@@ -19,6 +20,8 @@
     NotInitializedException,
     NotSupportedException,
 )
+from annif.util import parse_args
+from annif.vocab import SubjectIndexFilter
 
 if TYPE_CHECKING:
     from collections import defaultdict
@@ -56,6 +59,8 @@ class AnnifProject(DatadirMixin):
     _backend = None
     _vocab = None
     _vocab_lang = None
+    _vocab_kwargs = {}
+    _subject_index = None
     initialized = False
 
     # default values for configuration settings
@@ -188,9 +193,14 @@ def _initialize_vocab(self) -> None:
             raise ConfigurationException(
                 "vocab setting is missing", project_id=self.project_id
             )
-        self._vocab, self._vocab_lang = self.registry.get_vocab(
-            self.vocab_spec, self.language
-        )
+
+        match = re.match(r"([\w-]+)(\((.*)\))?$", self.vocab_spec)
+        if match is None:
+            raise ValueError(f"Invalid vocabulary specification: {self.vocab_spec}")
+        vocab_id = match.group(1)
+        posargs, self._vocab_kwargs = parse_args(match.group(3))
+        self._vocab_lang = posargs[0] if posargs else self.language
+        self._vocab = self.registry.get_vocab(vocab_id)
 
     @property
     def vocab(self) -> AnnifVocabulary:
@@ -206,7 +216,14 @@ def vocab_lang(self) -> str:
 
     @property
     def subjects(self) -> SubjectIndex:
-        return self.vocab.subjects
+        if self._subject_index is None:
+            self._subject_index = self.vocab.subjects
+            if "exclude" in self._vocab_kwargs:
+                exclude_list = self._vocab_kwargs["exclude"].split("|")
+                self._subject_index = SubjectIndexFilter(
+                    self._subject_index, exclude=exclude_list
+                )
+        return self._subject_index
 
     def _get_info(self, key: str) -> bool | datetime | None:
         try:
diff --git a/annif/registry.py b/annif/registry.py
index 01a5b9c82..07b7288f9 100644
--- a/annif/registry.py
+++ b/annif/registry.py
@@ -11,7 +11,6 @@
 from annif.config import parse_config
 from annif.exception import ConfigurationException
 from annif.project import Access, AnnifProject
-from annif.util import parse_args
 from annif.vocab import AnnifVocabulary
 
 logger = annif.logger
@@ -90,24 +89,16 @@ def get_project(
         except KeyError:
             raise ValueError("No such project {}".format(project_id))
 
-    def get_vocab(
-        self, vocab_spec: str, default_language: str | None
-    ) -> tuple[AnnifVocabulary, None] | tuple[AnnifVocabulary, str]:
-        """Return an (AnnifVocabulary, language) pair corresponding to the
-        vocab_spec. If no language information is specified, use the given
-        default language."""
+    def get_vocab(self, vocab_id: str) -> AnnifVocabulary:
+        """Return an AnnifVocabulary by vocab_id"""
 
-        match = re.match(r"([\w-]+)(\((.*)\))?$", vocab_spec)
-        if match is None:
-            raise ValueError(f"Invalid vocabulary specification: {vocab_spec}")
-        vocab_id = match.group(1)
-        posargs, kwargs = parse_args(match.group(3))
-        language = posargs[0] if posargs else default_language
+        if not vocab_id:
+            raise ValueError(f"Invalid vocabulary ID: '{vocab_id}'")
 
         self._init_vars()
         if vocab_id not in self._vocabs[self._rid]:
             self._vocabs[self._rid][vocab_id] = AnnifVocabulary(vocab_id, self._datadir)
-        return self._vocabs[self._rid][vocab_id], language
+        return self._vocabs[self._rid][vocab_id]
 
 
 def initialize_projects(app: Flask) -> None:
diff --git a/annif/suggestion.py b/annif/suggestion.py
index fd9eb48cc..fdbee9c80 100644
--- a/annif/suggestion.py
+++ b/annif/suggestion.py
@@ -95,11 +95,14 @@ def from_sequence(
         """Create a new SuggestionBatch from a sequence where each item is
         a sequence of SubjectSuggestion objects."""
 
-        deprecated = set(subject_index.deprecated_ids())
         data, rows, cols = [], [], []
         for idx, result in enumerate(suggestion_results):
             for suggestion in itertools.islice(result, limit):
-                if suggestion.subject_id in deprecated or suggestion.score <= 0.0:
+                if suggestion.score <= 0.0:
+                    continue
+                try:  # check for deprecated subjects
+                    _ = subject_index[suggestion.subject_id]
+                except IndexError:
                     continue
                 data.append(min(suggestion.score, 1.0))
                 rows.append(idx)
diff --git a/annif/vocab/__init__.py b/annif/vocab/__init__.py
new file mode 100644
index 000000000..6638e2331
--- /dev/null
+++ b/annif/vocab/__init__.py
@@ -0,0 +1,19 @@
+"""Annif vocabulary functionality"""
+
+from .skos import VocabFileSKOS
+from .subject_file import VocabFileCSV, VocabFileTSV
+from .subject_index import SubjectIndexFile, SubjectIndexFilter
+from .types import Subject, SubjectIndex, VocabSource
+from .vocab import AnnifVocabulary
+
+__all__ = [
+    "AnnifVocabulary",
+    "Subject",
+    "SubjectIndex",
+    "SubjectIndexFile",
+    "SubjectIndexFilter",
+    "VocabFileCSV",
+    "VocabFileSKOS",
+    "VocabFileTSV",
+    "VocabSource",
+]
diff --git a/annif/corpus/skos.py b/annif/vocab/skos.py
similarity index 97%
rename from annif/corpus/skos.py
rename to annif/vocab/skos.py
index 82a8c5c13..b3ea4e2e5 100644
--- a/annif/corpus/skos.py
+++ b/annif/vocab/skos.py
@@ -1,4 +1,4 @@
-"""Support for subjects loaded from a SKOS/RDF file"""
+"""Classes for supporting vocabulary files in SKOS/RDF format"""
 
 from __future__ import annotations
 
@@ -13,7 +13,7 @@
 
 import annif.util
 
-from .types import Subject, SubjectCorpus
+from .types import Subject, VocabSource
 
 if TYPE_CHECKING:
     from collections.abc import Iterator, Sequence
@@ -52,7 +52,7 @@ def serialize_subjects_to_skos(subjects: Iterator, path: str) -> None:
     )
 
 
-class SubjectFileSKOS(SubjectCorpus):
+class VocabFileSKOS(VocabSource):
     """A subject corpus that uses SKOS files"""
 
     PREF_LABEL_PROPERTIES = (SKOS.prefLabel, RDFS.label)
diff --git a/annif/vocab/subject_file.py b/annif/vocab/subject_file.py
new file mode 100644
index 000000000..253769878
--- /dev/null
+++ b/annif/vocab/subject_file.py
@@ -0,0 +1,107 @@
+"""Classes for supporting vocabulary files in CSV or TSV format"""
+
+from __future__ import annotations
+
+import csv
+import os.path
+from typing import TYPE_CHECKING
+
+import annif
+import annif.util
+
+from .skos import serialize_subjects_to_skos
+from .types import Subject, VocabSource
+
+if TYPE_CHECKING:
+    from collections.abc import Generator, Iterator
+
+
+class VocabFileTSV(VocabSource):
+    """A monolingual subject vocabulary stored in a TSV file."""
+
+    def __init__(self, path: str, language: str) -> None:
+        """initialize the VocabFileTSV given a path to a TSV file and the
+        language of the vocabulary"""
+
+        self.path = path
+        self.language = language
+
+    def _parse_line(self, line: str) -> Iterator[Subject]:
+        vals = line.strip().split("\t", 2)
+        clean_uri = annif.util.cleanup_uri(vals[0])
+        label = vals[1] if len(vals) >= 2 else None
+        labels = {self.language: label} if label else None
+        notation = vals[2] if len(vals) >= 3 else None
+        yield Subject(uri=clean_uri, labels=labels, notation=notation)
+
+    @property
+    def languages(self) -> list[str]:
+        return [self.language]
+
+    @property
+    def subjects(self) -> Generator:
+        with open(self.path, encoding="utf-8-sig") as subjfile:
+            for line in subjfile:
+                yield from self._parse_line(line)
+
+    def save_skos(self, path: str) -> None:
+        """Save the contents of the subject vocabulary into a SKOS/Turtle
+        file with the given path name."""
+        serialize_subjects_to_skos(self.subjects, path)
+
+
+class VocabFileCSV(VocabSource):
+    """A multilingual subject vocabulary stored in a CSV file."""
+
+    def __init__(self, path: str) -> None:
+        """initialize the VocabFileCSV given a path to a CSV file"""
+        self.path = path
+
+    def _parse_row(self, row: dict[str, str]) -> Iterator[Subject]:
+        labels = {
+            fname.replace("label_", ""): value or None
+            for fname, value in row.items()
+            if fname.startswith("label_")
+        }
+
+        # if there are no labels in any language, set labels to None
+        # indicating a deprecated subject
+        if set(labels.values()) == {None}:
+            labels = None
+
+        yield Subject(
+            uri=annif.util.cleanup_uri(row["uri"]),
+            labels=labels,
+            notation=row.get("notation", None) or None,
+        )
+
+    @property
+    def languages(self) -> list[str]:
+        # infer the supported languages from the CSV column names
+        with open(self.path, encoding="utf-8-sig") as csvfile:
+            reader = csv.reader(csvfile)
+            fieldnames = next(reader, None)
+
+        return [
+            fname.replace("label_", "")
+            for fname in fieldnames
+            if fname.startswith("label_")
+        ]
+
+    @property
+    def subjects(self) -> Generator:
+        with open(self.path, encoding="utf-8-sig") as csvfile:
+            reader = csv.DictReader(csvfile)
+            for row in reader:
+                yield from self._parse_row(row)
+
+    def save_skos(self, path: str) -> None:
+        """Save the contents of the subject vocabulary into a SKOS/Turtle
+        file with the given path name."""
+        serialize_subjects_to_skos(self.subjects, path)
+
+    @staticmethod
+    def is_csv_file(path: str) -> bool:
+        """return True if the path looks like a CSV file"""
+
+        return os.path.splitext(path)[1].lower() == ".csv"
diff --git a/annif/vocab/subject_index.py b/annif/vocab/subject_index.py
new file mode 100644
index 000000000..72b51a156
--- /dev/null
+++ b/annif/vocab/subject_index.py
@@ -0,0 +1,164 @@
+"""Subject index functionality for Annif"""
+
+from __future__ import annotations
+
+import csv
+
+import annif
+import annif.util
+
+from .subject_file import VocabFileCSV
+from .types import Subject, SubjectIndex, VocabSource
+
+logger = annif.logger
+logger.addFilter(annif.util.DuplicateFilter())
+
+
+class SubjectIndexFile(SubjectIndex):
+    """SubjectIndex implementation backed by a file."""
+
+    def __init__(self) -> None:
+        self._subjects = []
+        self._uri_idx = {}
+        self._label_idx = {}
+        self._languages = None
+
+    def load_subjects(self, vocab_source: VocabSource) -> None:
+        """Initialize the subject index from a subject corpus"""
+
+        self._languages = vocab_source.languages
+        for subject in vocab_source.subjects:
+            self.append(subject)
+
+    def __len__(self) -> int:
+        return len(self._subjects)
+
+    @property
+    def languages(self) -> list[str] | None:
+        return self._languages
+
+    def __getitem__(self, subject_id: int) -> Subject:
+        subject = self._subjects[subject_id]
+        if subject.labels is None:
+            raise IndexError(f"Subject is deprecated: {subject_id}")
+        return subject
+
+    def append(self, subject: Subject) -> None:
+        if self._languages is None and subject.labels is not None:
+            self._languages = list(subject.labels.keys())
+
+        subject_id = len(self._subjects)
+        self._uri_idx[subject.uri] = subject_id
+        if subject.labels:
+            for lang, label in subject.labels.items():
+                self._label_idx[(label, lang)] = subject_id
+        self._subjects.append(subject)
+
+    def contains_uri(self, uri: str) -> bool:
+        return uri in self._uri_idx
+
+    def by_uri(self, uri: str, warnings: bool = True) -> int | None:
+        try:
+            subject_id = self._uri_idx[uri]
+            if self._subjects[subject_id].labels is None:  # deprecated
+                return None
+            return subject_id
+        except KeyError:
+            if warnings:
+                logger.warning("Unknown subject URI <%s>", uri)
+            return None
+
+    def by_label(self, label: str | None, language: str) -> int | None:
+        try:
+            return self._label_idx[(label, language)]
+        except KeyError:
+            logger.warning('Unknown subject label "%s"@%s', label, language)
+            return None
+
+    @property
+    def active(self) -> list[tuple[int, Subject]]:
+        return [
+            (subj_id, subject)
+            for subj_id, subject in enumerate(self._subjects)
+            if subject.labels is not None
+        ]
+
+    def save(self, path: str) -> None:
+        """Save this subject index into a file with the given path name."""
+
+        fieldnames = ["uri", "notation"] + [f"label_{lang}" for lang in self._languages]
+
+        with open(path, "w", encoding="utf-8", newline="") as csvfile:
+            writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
+            writer.writeheader()
+            for subject in self:
+                row = {"uri": subject.uri, "notation": subject.notation or ""}
+                if subject.labels:
+                    for lang, label in subject.labels.items():
+                        row[f"label_{lang}"] = label
+                writer.writerow(row)
+
+    @classmethod
+    def load(cls, path: str) -> SubjectIndex:
+        """Load a subject index from a CSV file and return it."""
+
+        vocab_file = VocabFileCSV(path)
+        subject_index = cls()
+        subject_index.load_subjects(vocab_file)
+        return subject_index
+
+
+class SubjectIndexFilter(SubjectIndex):
+    """SubjectIndex implementation that filters another SubjectIndex based
+    on a list of subject URIs to exclude."""
+
+    def __init__(self, subject_index: SubjectIndex, exclude: list[str]):
+        self._subject_index = subject_index
+        self._exclude = set(exclude)
+
+    def __len__(self) -> int:
+        return len(self._subject_index)
+
+    @property
+    def languages(self) -> list[str] | None:
+        return self._subject_index.languages
+
+    def __getitem__(self, subject_id: int) -> Subject:
+        subject = self._subject_index[subject_id]
+        if subject.uri in self._exclude:
+            raise IndexError(f"Subject is excluded: {subject.uri}")
+        return subject
+
+    def contains_uri(self, uri: str) -> bool:
+        if uri in self._exclude:
+            return False
+        return self._subject_index.contains_uri(uri)
+
+    def by_uri(self, uri: str, warnings: bool = True) -> int | None:
+        """return the subject ID of a subject by its URI, or None if not found.
+        If warnings=True, log a warning message if the URI cannot be found."""
+
+        if uri in self._exclude:
+            return None
+        return self._subject_index.by_uri(uri, warnings)
+
+    def by_label(self, label: str | None, language: str) -> int | None:
+        """return the subject ID of a subject by its label in a given
+        language"""
+
+        subject_id = self._subject_index.by_label(label, language)
+        subject = self._subject_index[subject_id]
+        if subject is not None and subject.uri not in self._exclude:
+            return subject_id
+        return None
+
+    @property
+    def active(self) -> list[tuple[int, Subject]]:
+        """return a list of (subject_id, Subject) tuples of all subjects that
+        are available for use"""
+
+        return [
+            (subject_id, subject)
+            for subject_id, subject in self._subject_index.active
+            if subject.uri not in self._exclude
+        ]
diff --git a/annif/vocab/types.py b/annif/vocab/types.py
new file mode 100644
index 000000000..8e80a35ca
--- /dev/null
+++ b/annif/vocab/types.py
@@ -0,0 +1,70 @@
+"""Type declarations for vocabulary functionality"""
+
+from __future__ import annotations
+
+import abc
+import collections
+
+Subject = collections.namedtuple("Subject", "uri labels notation")
+
+
+class VocabSource(metaclass=abc.ABCMeta):
+    """Abstract base class for vocabulary sources"""
+
+    @property
+    @abc.abstractmethod
+    def subjects(self):
+        """Iterate through the vocabulary, yielding Subject objects."""
+        pass  # pragma: no cover
+
+    @property
+    @abc.abstractmethod
+    def languages(self):
+        """Provide a list of language codes supported by this vocabulary source."""
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def save_skos(self, path):
+        """Save the contents of the vocabulary source into a SKOS/Turtle
+        file with the given path name."""
+        pass  # pragma: no cover
+
+
+class SubjectIndex(metaclass=abc.ABCMeta):
+    """Base class for an index that remembers the associations between
+    integer subject IDs and their URIs and labels."""
+
+    @abc.abstractmethod
+    def __len__(self) -> int:
+        pass  # pragma: no cover
+
+    @property
+    @abc.abstractmethod
+    def languages(self) -> list[str] | None:
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def __getitem__(self, subject_id: int) -> Subject:
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def contains_uri(self, uri: str) -> bool:
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def by_uri(self, uri: str, warnings: bool = True) -> int | None:
+        """return the subject ID of a subject by its URI, or None if not found.
+        If warnings=True, log a warning message if the URI cannot be found."""
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def by_label(self, label: str | None, language: str) -> int | None:
+        """return the subject ID of a subject by its label in a given
+        language"""
+        pass  # pragma: no cover
+
+    @abc.abstractmethod
+    def active(self) -> list[tuple[int, Subject]]:
+        """return a list of (subject_id, Subject) tuples of all subjects that
+        are available for use"""
+        pass  # pragma: no cover
diff --git a/annif/vocab.py b/annif/vocab/vocab.py
similarity index 80%
rename from annif/vocab.py
rename to annif/vocab/vocab.py
index 624b44403..a297f7ec8 100644
--- a/annif/vocab.py
+++ b/annif/vocab/vocab.py
@@ -11,14 +11,16 @@
 from annif.datadir import DatadirMixin
 from annif.exception import NotInitializedException
 
+from .skos import VocabFileSKOS
+from .subject_index import SubjectIndexFile
+from .types import Subject, SubjectIndex, VocabSource
+
 if TYPE_CHECKING:
     from rdflib.graph import Graph
 
-    from annif.corpus.skos import SubjectFileSKOS
-    from annif.corpus.subject import SubjectCorpus, SubjectIndex
-
 
 logger = annif.logger
+logger.addFilter(annif.util.DuplicateFilter())
 
 
 class AnnifVocabulary(DatadirMixin):
@@ -38,25 +40,23 @@ def __init__(self, vocab_id: str, datadir: str) -> None:
         self.vocab_id = vocab_id
         self._skos_vocab = None
 
-    def _create_subject_index(self, subject_corpus: SubjectCorpus) -> SubjectIndex:
-        subjects = annif.corpus.SubjectIndex()
-        subjects.load_subjects(subject_corpus)
+    def _create_subject_index(self, vocab_source: VocabSource) -> SubjectIndex:
+        subjects = SubjectIndexFile()
+        subjects.load_subjects(vocab_source)
         annif.util.atomic_save(subjects, self.datadir, self.INDEX_FILENAME_CSV)
         return subjects
 
-    def _update_subject_index(self, subject_corpus: SubjectCorpus) -> SubjectIndex:
+    def _update_subject_index(self, vocab_source: VocabSource) -> SubjectIndex:
         old_subjects = self.subjects
-        new_subjects = annif.corpus.SubjectIndex()
-        new_subjects.load_subjects(subject_corpus)
-        updated_subjects = annif.corpus.SubjectIndex()
+        new_subjects = SubjectIndexFile()
+        new_subjects.load_subjects(vocab_source)
+        updated_subjects = SubjectIndexFile()
 
         for old_subject in old_subjects:
             if new_subjects.contains_uri(old_subject.uri):
                 new_subject = new_subjects[new_subjects.by_uri(old_subject.uri)]
             else:  # subject removed from new corpus
-                new_subject = annif.corpus.Subject(
-                    uri=old_subject.uri, labels=None, notation=None
-                )
+                new_subject = Subject(uri=old_subject.uri, labels=None, notation=None)
             updated_subjects.append(new_subject)
         for new_subject in new_subjects:
             if not old_subjects.contains_uri(new_subject.uri):
@@ -70,13 +70,13 @@ def subjects(self) -> SubjectIndex:
             path = os.path.join(self.datadir, self.INDEX_FILENAME_CSV)
             if os.path.exists(path):
                 logger.debug("loading subjects from %s", path)
-                self._subjects = annif.corpus.SubjectIndex.load(path)
+                self._subjects = SubjectIndexFile.load(path)
             else:
                 raise NotInitializedException("subject file {} not found".format(path))
         return self._subjects
 
     @property
-    def skos(self) -> SubjectFileSKOS:
+    def skos(self) -> VocabFileSKOS:
         """return the subject vocabulary from SKOS file"""
         if self._skos_vocab is not None:
             return self._skos_vocab
@@ -86,7 +86,7 @@ def skos(self) -> SubjectFileSKOS:
         if os.path.exists(dumppath):
             logger.debug(f"loading graph dump from {dumppath}")
             try:
-                self._skos_vocab = annif.corpus.SubjectFileSKOS(dumppath)
+                self._skos_vocab = VocabFileSKOS(dumppath)
             except ModuleNotFoundError:
                 # Probably dump has been saved using a different rdflib version
                 logger.debug("could not load graph dump, using turtle file")
@@ -97,7 +97,7 @@ def skos(self) -> SubjectFileSKOS:
         path = os.path.join(self.datadir, self.INDEX_FILENAME_TTL)
         if os.path.exists(path):
             logger.debug(f"loading graph from {path}")
-            self._skos_vocab = annif.corpus.SubjectFileSKOS(path)
+            self._skos_vocab = VocabFileSKOS(path)
             # store the dump file so we can use it next time
             self._skos_vocab.save_skos(path)
             return self._skos_vocab
@@ -116,7 +116,7 @@ def languages(self) -> list[str]:
 
     def load_vocabulary(
         self,
-        subject_corpus: SubjectCorpus,
+        vocab_source: VocabSource,
         force: bool = False,
     ) -> None:
         """Load subjects from a subject corpus and save them into one
@@ -127,14 +127,14 @@ def load_vocabulary(
             os.path.join(self.datadir, self.INDEX_FILENAME_CSV)
         ):
             logger.info("updating existing subject index")
-            self._subjects = self._update_subject_index(subject_corpus)
+            self._subjects = self._update_subject_index(vocab_source)
         else:
             logger.info("creating subject index")
-            self._subjects = self._create_subject_index(subject_corpus)
+            self._subjects = self._create_subject_index(vocab_source)
 
         skosfile = os.path.join(self.datadir, self.INDEX_FILENAME_TTL)
         logger.info(f"saving vocabulary into SKOS file {skosfile}")
-        subject_corpus.save_skos(skosfile)
+        vocab_source.save_skos(skosfile)
 
     def as_graph(self) -> Graph:
         """return the vocabulary as an rdflib graph"""
diff --git a/tests/conftest.py b/tests/conftest.py
index 9f2015a31..8dd81514d 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -12,6 +12,7 @@
 import annif.corpus
 import annif.project
 import annif.registry
+import annif.vocab
 
 
 @pytest.fixture(scope="module")
@@ -22,7 +23,7 @@ def cxapp():
     with cxapp.app.app_context():
         project = annif.registry.get_project("dummy-en")
         # the vocab is needed for both English and Finnish language projects
-        vocab = annif.corpus.SubjectFileCSV(subjfile)
+        vocab = annif.vocab.VocabFileCSV(subjfile)
         project.vocab.load_vocabulary(vocab)
     return cxapp
 
@@ -78,7 +79,7 @@ def subject_file():
     docfile = os.path.join(
         os.path.dirname(__file__), "corpora", "archaeology", "subjects.tsv"
     )
-    return annif.corpus.SubjectFileTSV(docfile, "fi")
+    return annif.vocab.VocabFileTSV(docfile, "fi")
 
 
 @pytest.fixture(scope="module")
@@ -94,7 +95,7 @@ def vocabulary(datadir):
     subjfile = os.path.join(
         os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
     )
-    subjects = annif.corpus.SubjectFileSKOS(subjfile)
+    subjects = annif.vocab.VocabFileSKOS(subjfile)
     vocab.load_vocabulary(subjects)
     return vocab
 
diff --git a/tests/projects.cfg b/tests/projects.cfg
index 5f7ce793c..9c6bbe7aa 100644
--- a/tests/projects.cfg
+++ b/tests/projects.cfg
@@ -32,6 +32,13 @@ backend=dummy
 analyzer=snowball(english)
 vocab=dummy(fi)
 
+[dummy-exclude]
+name=Dummy with an excluded concept
+language=en
+backend=dummy
+analyzer=snowball(english)
+vocab=dummy(en,exclude=http://example.org/dummy)
+
 [dummy-transform]
 name=Dummy with pass-through transform
 language=fi
diff --git a/tests/test_backend_http.py b/tests/test_backend_http.py
index 114bcbe96..0b8c3a37f 100644
--- a/tests/test_backend_http.py
+++ b/tests/test_backend_http.py
@@ -8,8 +8,8 @@
 import requests.exceptions
 
 import annif.backend.http
-from annif.corpus import Subject
 from annif.exception import OperationFailedException
+from annif.vocab import Subject
 
 
 def test_http_suggest(app_project):
diff --git a/tests/test_cli.py b/tests/test_cli.py
index 5690be484..34d2440b1 100644
--- a/tests/test_cli.py
+++ b/tests/test_cli.py
@@ -297,7 +297,7 @@ def test_load_vocab_nonexistent_path():
     assert failed_result.exception
     assert failed_result.exit_code != 0
     assert (
-        "Invalid value for 'SUBJECTFILE': "
+        "Invalid value for 'VOCAB_FILE': "
         "File 'nonexistent_path' does not exist." in failed_result.output
     )
 
@@ -455,6 +455,15 @@ def test_suggest_with_different_vocab_language():
     assert result.exit_code == 0
 
 
+def test_suggest_with_exclude():
+    result = runner.invoke(
+        annif.cli.cli, ["suggest", "dummy-exclude"], input="the cat sat on the mat"
+    )
+    assert not result.exception
+    assert result.output == ""  # nothing returned, the concept was excluded
+    assert result.exit_code == 0
+
+
 def test_suggest_with_notations():
     result = runner.invoke(
         annif.cli.cli,
@@ -1124,7 +1133,7 @@ def test_upload_many(
     result = runner.invoke(annif.cli.cli, ["upload", "dummy-*", "dummy-repo"])
     assert not result.exception
     assert create_commit.call_count == 1
-    assert CommitOperationAdd.call_count == 11
+    assert CommitOperationAdd.call_count == 13
     assert upsert_modelcard.call_count == 1
 
 
@@ -1411,6 +1420,7 @@ def test_completion_show_project_project_ids_all():
         "dummy-en",
         "dummy-private",
         "dummy-vocablang",
+        "dummy-exclude",
         "dummy-transform",
         "limit-transform",
         "ensemble",
@@ -1436,6 +1446,7 @@ def test_completion_show_project_project_ids_dummy():
         "dummy-en",
         "dummy-private",
         "dummy-vocablang",
+        "dummy-exclude",
         "dummy-transform",
     ]
 
diff --git a/tests/test_config.py b/tests/test_config.py
index 0cb50d0c3..816ba49c1 100644
--- a/tests/test_config.py
+++ b/tests/test_config.py
@@ -52,7 +52,7 @@ def test_find_config_not_exists_default(monkeypatch, caplog):
 def test_parse_config_cfg_nondefault():
     cfg = annif.config.parse_config("tests/projects.cfg")
     assert isinstance(cfg, annif.config.AnnifConfigCFG)
-    assert len(cfg.project_ids) == 18
+    assert len(cfg.project_ids) == 19
     assert cfg["dummy-fi"] is not None
 
 
@@ -61,7 +61,7 @@ def test_parse_config_cfg_default(monkeypatch):
     monkeypatch.chdir("tests")
     cfg = annif.config.parse_config("")
     assert isinstance(cfg, annif.config.AnnifConfigCFG)
-    assert len(cfg.project_ids) == 18
+    assert len(cfg.project_ids) == 19
     assert cfg["dummy-fi"] is not None
 
 
@@ -83,7 +83,7 @@ def test_parse_config_toml_failed(tmpdir):
 def test_parse_config_directory():
     cfg = annif.config.parse_config("tests/projects.d")
     assert isinstance(cfg, annif.config.AnnifConfigDirectory)
-    assert len(cfg.project_ids) == 2 + 18  # 0-projects.toml + 1-projects.cfg
+    assert len(cfg.project_ids) == 2 + 19  # 0-projects.toml + 1-projects.cfg
     assert cfg["dummy-fi"] is not None
     assert cfg["dummy-fi-toml"] is not None
     assert list(cfg.project_ids)[0] == "dummy-fi-toml"  # First in 0-projects.toml
diff --git a/tests/test_corpus.py b/tests/test_corpus.py
index 6e7db5158..f8cd7ee01 100644
--- a/tests/test_corpus.py
+++ b/tests/test_corpus.py
@@ -226,26 +226,6 @@ def test_docdir_key_as_doccorpus(tmpdir, subject_index):
     )
 
 
-def test_subject_by_uri(subject_index):
-    subj_id = subject_index.by_uri("http://www.yso.fi/onto/yso/p7141")
-    assert subject_index[subj_id].labels["fi"] == "sinetit"
-
-
-def test_subject_by_uri_missing(subject_index):
-    subj_id = subject_index.by_uri("http://nonexistent")
-    assert subj_id is None
-
-
-def test_subject_by_label(subject_index):
-    subj_id = subject_index.by_label("sinetit", "fi")
-    assert subject_index[subj_id].uri == "http://www.yso.fi/onto/yso/p7141"
-
-
-def test_subject_by_label_missing(subject_index):
-    subj_id = subject_index.by_label("nonexistent", "fi")
-    assert subj_id is None
-
-
 def test_docfile_plain(tmpdir, subject_index):
     docfile = tmpdir.join("documents.tsv")
     docfile.write(
diff --git a/tests/test_project.py b/tests/test_project.py
index 6fcd2c9af..40d052529 100644
--- a/tests/test_project.py
+++ b/tests/test_project.py
@@ -337,6 +337,6 @@ def test_project_file_toml():
 def test_project_directory():
     cxapp = annif.create_app(config_name="annif.default_config.TestingDirectoryConfig")
     with cxapp.app.app_context():
-        assert len(annif.registry.get_projects()) == 18 + 2
+        assert len(annif.registry.get_projects()) == 19 + 2
         assert annif.registry.get_project("dummy-fi").project_id == "dummy-fi"
         assert annif.registry.get_project("dummy-fi-toml").project_id == "dummy-fi-toml"
diff --git a/tests/test_rest.py b/tests/test_rest.py
index d6c2160c4..dd1ff7ceb 100644
--- a/tests/test_rest.py
+++ b/tests/test_rest.py
@@ -156,6 +156,14 @@ def test_rest_suggest_with_different_vocab_language(app):
         assert result["results"][0]["label"] == "dummy-fi"
 
 
+def test_rest_suggest_with_exclude(app):
+    with app.app_context():
+        result = annif.rest.suggest(
+            "dummy-exclude", {"text": "example text", "limit": 10, "threshold": 0.0}
+        )[0]
+        assert not result["results"]
+
+
 def test_rest_suggest_with_notations(app):
     with app.app_context():
         result = annif.rest.suggest(
diff --git a/tests/test_suggestion.py b/tests/test_suggestion.py
index 2155a7bd1..2b126bcbc 100644
--- a/tests/test_suggestion.py
+++ b/tests/test_suggestion.py
@@ -4,13 +4,13 @@
 import pytest
 from scipy.sparse import csr_array
 
-from annif.corpus import Subject
 from annif.suggestion import (
     SubjectSuggestion,
     SuggestionBatch,
     filter_suggestion,
     vector_to_suggestions,
 )
+from annif.vocab import Subject
 
 
 def test_vector_to_suggestions():
@@ -132,6 +132,7 @@ def test_suggestionbatch_from_sequence_with_deprecated(dummy_subject_index):
     dummy_subject_index.append(
         Subject(uri="http://example.org/deprecated", labels=None, notation=None)
     )
+    deprecated_id = len(dummy_subject_index) - 1
 
     orig_suggestions = [
         [
@@ -140,7 +141,7 @@ def test_suggestionbatch_from_sequence_with_deprecated(dummy_subject_index):
                 score=0.8,
             ),
             SubjectSuggestion(
-                subject_id=dummy_subject_index.by_uri("http://example.org/deprecated"),
+                subject_id=deprecated_id,
                 score=0.5,
             ),
             SubjectSuggestion(
diff --git a/tests/test_vocab.py b/tests/test_vocab.py
index b2b51d3c0..504635efe 100644
--- a/tests/test_vocab.py
+++ b/tests/test_vocab.py
@@ -12,20 +12,22 @@
 
 def load_dummy_vocab(tmpdir):
     vocab = annif.vocab.AnnifVocabulary("vocab-id", str(tmpdir))
-    subjfile = os.path.join(os.path.dirname(__file__), "corpora", "dummy-subjects.tsv")
-    subjects = annif.corpus.SubjectFileTSV(subjfile, "en")
-    vocab.load_vocabulary(subjects)
+    vocab_path = os.path.join(
+        os.path.dirname(__file__), "corpora", "dummy-subjects.tsv"
+    )
+    vocab_file = annif.vocab.VocabFileTSV(vocab_path, "en")
+    vocab.load_vocabulary(vocab_file)
     return vocab
 
 
 def test_get_vocab_invalid(registry):
     with pytest.raises(ValueError) as excinfo:
-        registry.get_vocab("", None)
-    assert "Invalid vocabulary specification" in str(excinfo.value)
+        registry.get_vocab("")
+    assert "Invalid vocabulary ID" in str(excinfo.value)
 
 
 def test_get_vocab_hyphen(registry):
-    vocab, lang = registry.get_vocab("dummy-noname", None)
+    vocab = registry.get_vocab("dummy-noname")
     assert vocab.vocab_id == "dummy-noname"
     assert vocab is not None
 
@@ -33,10 +35,12 @@ def test_get_vocab_hyphen(registry):
 def test_update_subject_index_with_no_changes(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile = os.path.join(os.path.dirname(__file__), "corpora", "dummy-subjects.tsv")
-    subjects = annif.corpus.SubjectFileTSV(subjfile, "en")
+    vocab_path = os.path.join(
+        os.path.dirname(__file__), "corpora", "dummy-subjects.tsv"
+    )
+    vocab_file = annif.vocab.VocabFileTSV(vocab_path, "en")
 
-    vocab.load_vocabulary(subjects)
+    vocab.load_vocabulary(vocab_file)
     assert len(vocab.subjects) == 2
     assert vocab.subjects.by_uri("http://example.org/dummy") == 0
     assert vocab.subjects[0].uri == "http://example.org/dummy"
@@ -51,33 +55,30 @@ def test_update_subject_index_with_no_changes(tmpdir):
 def test_update_subject_index_with_removed_subject(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile_new = tmpdir.join("subjects_new.tsv")
-    subjfile_new.write("<http://example.org/dummy>\tdummy\n")
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
+    voctmp_new = tmpdir.join("subjects_new.tsv")
+    voctmp_new.write("<http://example.org/dummy>\tdummy\n")
+    vocab_file_new = annif.vocab.VocabFileTSV(str(voctmp_new), "en")
 
-    vocab.load_vocabulary(subjects_new)
+    vocab.load_vocabulary(vocab_file_new)
     assert len(vocab.subjects) == 2
     assert vocab.subjects.by_uri("http://example.org/dummy") == 0
     assert vocab.subjects[0].uri == "http://example.org/dummy"
     assert vocab.subjects[0].labels["en"] == "dummy"
     assert vocab.subjects[0].notation is None
-    assert vocab.subjects.by_uri("http://example.org/none") == 1
-    assert vocab.subjects[1].uri == "http://example.org/none"
-    assert vocab.subjects[1].labels is None
-    assert vocab.subjects[1].notation is None
+    assert vocab.subjects.by_uri("http://example.org/none") is None
 
 
 def test_update_subject_index_with_renamed_label_and_added_notation(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
 
-    subjfile_new = tmpdir.join("subjects_new.tsv")
-    subjfile_new.write(
+    voctmp_new = tmpdir.join("subjects_new.tsv")
+    voctmp_new.write(
         "<http://example.org/dummy>\tdummy\n"
         + "<http://example.org/none>\tnew none\t42.42\n"
     )
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
+    vocab_file_new = annif.vocab.VocabFileTSV(str(voctmp_new), "en")
 
-    vocab.load_vocabulary(subjects_new)
+    vocab.load_vocabulary(vocab_file_new)
     assert len(vocab.subjects) == 2
     assert vocab.subjects.by_uri("http://example.org/dummy") == 0
     assert vocab.subjects[0].uri == "http://example.org/dummy"
@@ -91,16 +92,16 @@ def test_update_subject_index_with_renamed_label_and_added_notation(tmpdir):
 
 def test_update_subject_index_with_added_subjects(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    subjfile_new = tmpdir.join("subjects_new.tsv")
-    subjfile_new.write(
+    voctmp_new = tmpdir.join("subjects_new.tsv")
+    voctmp_new.write(
         "<http://example.org/dummy>\tdummy\n"
         + "<http://example.org/none>\tnone\n"
         + "<http://example.org/new-dummy>\tnew dummy\t42.42\n"
         + "<http://example.org/new-none>\tnew none\n"
     )
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
+    vocab_file_new = annif.vocab.VocabFileTSV(str(voctmp_new), "en")
 
-    vocab.load_vocabulary(subjects_new)
+    vocab.load_vocabulary(vocab_file_new)
     assert len(vocab.subjects) == 4
     assert vocab.subjects.by_uri("http://example.org/dummy") == 0
     assert vocab.subjects[0].uri == "http://example.org/dummy"
@@ -114,15 +115,15 @@ def test_update_subject_index_with_added_subjects(tmpdir):
 
 def test_update_subject_index_force(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
-    subjfile_new = tmpdir.join("subjects_new.tsv")
-    subjfile_new.write(
+    voctmp_new = tmpdir.join("subjects_new.tsv")
+    voctmp_new.write(
         "<http://example.org/dummy>\tdummy\n"
         + "<http://example.org/new-dummy>\tnew dummy\t42.42\n"
         + "<http://example.org/new-none>\tnew none\n"
     )
-    subjects_new = annif.corpus.SubjectFileTSV(str(subjfile_new), "en")
+    vocab_file_new = annif.vocab.VocabFileTSV(str(voctmp_new), "en")
 
-    vocab.load_vocabulary(subjects_new, force=True)
+    vocab.load_vocabulary(vocab_file_new, force=True)
     assert len(vocab.subjects) == 3
     assert vocab.subjects.by_uri("http://example.org/dummy") == 0
     assert vocab.subjects[0].uri == "http://example.org/dummy"
@@ -138,7 +139,7 @@ def test_skos(tmpdir):
     vocab = load_dummy_vocab(tmpdir)
     assert tmpdir.join("vocabs/vocab-id/subjects.ttl").exists()
     assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
-    assert isinstance(vocab.skos, annif.corpus.SubjectFileSKOS)
+    assert isinstance(vocab.skos, annif.vocab.VocabFileSKOS)
 
 
 def test_skos_cache(tmpdir):
@@ -148,7 +149,7 @@ def test_skos_cache(tmpdir):
     tmpdir.join("vocabs/vocab-id/subjects.dump.gz").remove()
     assert not tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
 
-    assert isinstance(vocab.skos, annif.corpus.SubjectFileSKOS)
+    assert isinstance(vocab.skos, annif.vocab.VocabFileSKOS)
     # cached dump file has been recreated in .skos property access
     assert tmpdir.join("vocabs/vocab-id/subjects.dump.gz").exists()
 
@@ -180,3 +181,53 @@ def test_as_graph(tmpdir):
     assert len(concepts) == 2
     assert "http://example.org/dummy" in concepts
     assert "http://example.org/none" in concepts
+
+
+def test_subject_by_uri(subject_index):
+    subj_id = subject_index.by_uri("http://www.yso.fi/onto/yso/p7141")
+    assert subject_index[subj_id].labels["fi"] == "sinetit"
+
+
+def test_subject_by_uri_missing(subject_index):
+    subj_id = subject_index.by_uri("http://nonexistent")
+    assert subj_id is None
+
+
+def test_subject_by_label(subject_index):
+    subj_id = subject_index.by_label("sinetit", "fi")
+    assert subject_index[subj_id].uri == "http://www.yso.fi/onto/yso/p7141"
+
+
+def test_subject_by_label_missing(subject_index):
+    subj_id = subject_index.by_label("nonexistent", "fi")
+    assert subj_id is None
+
+
+def test_subject_index_filter(subject_index):
+    excluded_uri = "http://www.yso.fi/onto/yso/p7141"  # sinetit@fi
+    included_uri = "http://www.yso.fi/onto/yso/p1265"  # arkeologia@fi
+
+    subject_filter = annif.vocab.SubjectIndexFilter(
+        subject_index, exclude=[excluded_uri]
+    )
+
+    assert len(subject_index) == len(subject_filter)
+
+    assert subject_index.languages == subject_filter.languages
+
+    subj_id_exc = subject_index.by_uri(excluded_uri)
+    with pytest.raises(IndexError):
+        _ = subject_filter[subj_id_exc]
+    subj_id_inc = subject_index.by_uri(included_uri)
+    assert subject_filter[subj_id_inc] is not None
+
+    assert subject_filter.by_uri(excluded_uri) is None
+    assert subject_filter.by_uri(included_uri) is not None
+
+    assert not subject_filter.contains_uri(excluded_uri)
+    assert subject_filter.contains_uri(included_uri)
+
+    assert subject_filter.by_label("sinetit", "fi") is None
+    assert subject_filter.by_label("arkeologia", "fi") is not None
+
+    assert len(subject_filter.active) == len(subject_index.active) - 1
diff --git a/tests/test_vocab_csv.py b/tests/test_vocab_csv.py
index bdb66e25c..f9d536324 100644
--- a/tests/test_vocab_csv.py
+++ b/tests/test_vocab_csv.py
@@ -1,22 +1,22 @@
 """Unit tests for CSV vocabulary functionality in Annif"""
 
-from annif.corpus import SubjectFileCSV, SubjectIndex
+from annif.vocab import SubjectIndexFile, VocabFileCSV
 
 
 def test_recognize_csv_lowercase():
-    assert SubjectFileCSV.is_csv_file("subjects.csv")
+    assert VocabFileCSV.is_csv_file("subjects.csv")
 
 
 def test_recognize_csv_uppercase():
-    assert SubjectFileCSV.is_csv_file("SUBJECTS.CSV")
+    assert VocabFileCSV.is_csv_file("SUBJECTS.CSV")
 
 
 def test_recognize_tsv():
-    assert not SubjectFileCSV.is_csv_file("subjects.tsv")
+    assert not VocabFileCSV.is_csv_file("subjects.tsv")
 
 
 def test_recognize_noext():
-    assert not SubjectFileCSV.is_csv_file("subjects")
+    assert not VocabFileCSV.is_csv_file("subjects")
 
 
 def test_load_csv_uri_brackets(tmpdir):
@@ -27,8 +27,8 @@ def test_load_csv_uri_brackets(tmpdir):
         + "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi"
     )
 
-    corpus = SubjectFileCSV(str(tmpfile))
-    subjects = list(corpus.subjects)
+    vocab_file = VocabFileCSV(str(tmpfile))
+    subjects = list(vocab_file.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
     assert subjects[0].labels["fi"] == "hylyt"
@@ -46,8 +46,8 @@ def test_load_tsv_uri_nobrackets(tmpdir):
         + "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi"
     )
 
-    corpus = SubjectFileCSV(str(tmpfile))
-    subjects = list(corpus.subjects)
+    vocab_file = VocabFileCSV(str(tmpfile))
+    subjects = list(vocab_file.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
     assert subjects[0].labels["fi"] == "hylyt"
@@ -65,8 +65,8 @@ def test_load_tsv_with_notations(tmpdir):
         + "http://www.yso.fi/onto/yso/p9285,neoliittinen kausi,42.0"
     )
 
-    corpus = SubjectFileCSV(str(tmpfile))
-    subjects = list(corpus.subjects)
+    vocab_file = VocabFileCSV(str(tmpfile))
+    subjects = list(vocab_file.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
     assert subjects[0].labels["fi"] == "hylyt"
@@ -85,13 +85,13 @@ def test_load_tsv_with_deprecated(tmpdir):
         + "<http://www.yso.fi/onto/yso/p9285>,neoliittinen kausi"
     )
 
-    corpus = SubjectFileCSV(str(tmpfile))
-    subjects = list(corpus.subjects)
-    assert len(list(corpus.subjects)) == 3
+    vocab_file = VocabFileCSV(str(tmpfile))
+    subjects = list(vocab_file.subjects)
+    assert len(list(vocab_file.subjects)) == 3
     assert subjects[1].labels is None
 
-    index = SubjectIndex()
-    index.load_subjects(corpus)
+    index = SubjectIndexFile()
+    index.load_subjects(vocab_file)
 
     active = list(index.active)
     assert len(active) == 2
diff --git a/tests/test_vocab_skos.py b/tests/test_vocab_skos.py
index fd93ca6d1..41f840ea9 100644
--- a/tests/test_vocab_skos.py
+++ b/tests/test_vocab_skos.py
@@ -2,27 +2,27 @@
 
 import os.path
 
-from annif.corpus.skos import SubjectFileSKOS
+from annif.vocab import VocabFileSKOS
 
 
 def test_recognize_turtle():
-    assert SubjectFileSKOS.is_rdf_file("subjects.ttl")
+    assert VocabFileSKOS.is_rdf_file("subjects.ttl")
 
 
 def test_recognize_rdfxml():
-    assert SubjectFileSKOS.is_rdf_file("subjects.rdf")
+    assert VocabFileSKOS.is_rdf_file("subjects.rdf")
 
 
 def test_recognize_nt():
-    assert SubjectFileSKOS.is_rdf_file("subjects.nt")
+    assert VocabFileSKOS.is_rdf_file("subjects.nt")
 
 
 def test_recognize_tsv():
-    assert not SubjectFileSKOS.is_rdf_file("subjects.tsv")
+    assert not VocabFileSKOS.is_rdf_file("subjects.tsv")
 
 
 def test_recognize_noext():
-    assert not SubjectFileSKOS.is_rdf_file("subjects")
+    assert not VocabFileSKOS.is_rdf_file("subjects")
 
 
 def test_load_turtle(tmpdir):
@@ -47,7 +47,7 @@ def test_load_turtle(tmpdir):
     """
     )
 
-    corpus = SubjectFileSKOS(str(tmpfile))
+    corpus = VocabFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 1  # one of the concepts was deprecated
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
@@ -71,7 +71,7 @@ def test_load_turtle_with_notation(tmpdir):
     """
     )
 
-    corpus = SubjectFileSKOS(str(tmpfile))
+    corpus = VocabFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
     assert subjects[0].labels["fi"] == "hylyt"
@@ -98,7 +98,7 @@ def test_load_turtle_missing_langtags(tmpdir):
     """
     )
 
-    corpus = SubjectFileSKOS(str(tmpfile))
+    corpus = VocabFileSKOS(str(tmpfile))
     subjects = list(corpus.subjects)
     assert len(subjects) == 3
 
@@ -113,7 +113,7 @@ def test_load_turtle_get_languages(testdatadir):
     subjectfile = os.path.join(
         os.path.dirname(__file__), "corpora", "archaeology", "yso-archaeology.ttl"
     )
-    corpus = SubjectFileSKOS(subjectfile)
+    corpus = VocabFileSKOS(subjectfile)
     langs = corpus.languages
     assert len(langs) == 3
     assert "fi" in langs
diff --git a/tests/test_vocab_tsv.py b/tests/test_vocab_tsv.py
index 4d0956d5a..d7ec7fd10 100644
--- a/tests/test_vocab_tsv.py
+++ b/tests/test_vocab_tsv.py
@@ -1,6 +1,6 @@
 """Unit tests for TSV vocabulary functionality in Annif"""
 
-from annif.corpus import SubjectFileTSV, SubjectIndex
+from annif.vocab import SubjectIndexFile, VocabFileTSV
 
 
 def test_load_tsv_uri_brackets(tmpdir):
@@ -10,7 +10,7 @@ def test_load_tsv_uri_brackets(tmpdir):
         + "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi"
     )
 
-    corpus = SubjectFileTSV(str(tmpfile), "fi")
+    corpus = VocabFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
@@ -28,7 +28,7 @@ def test_load_tsv_uri_nobrackets(tmpdir):
         + "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi"
     )
 
-    corpus = SubjectFileTSV(str(tmpfile), "fi")
+    corpus = VocabFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
@@ -46,7 +46,7 @@ def test_load_tsv_with_notations(tmpdir):
         + "http://www.yso.fi/onto/yso/p9285\tneoliittinen kausi\t42.0"
     )
 
-    corpus = SubjectFileTSV(str(tmpfile), "fi")
+    corpus = VocabFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(subjects) == 2
     assert subjects[0].uri == "http://www.yso.fi/onto/yso/p8993"
@@ -65,12 +65,12 @@ def test_load_tsv_with_deprecated(tmpdir):
         + "<http://www.yso.fi/onto/yso/p9285>\tneoliittinen kausi"
     )
 
-    corpus = SubjectFileTSV(str(tmpfile), "fi")
+    corpus = VocabFileTSV(str(tmpfile), "fi")
     subjects = list(corpus.subjects)
     assert len(list(corpus.subjects)) == 3
     assert subjects[1].labels is None
 
-    index = SubjectIndex()
+    index = SubjectIndexFile()
     index.load_subjects(corpus)
 
     active = list(index.active)


Reviews:
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-26T08:21:29Z
user: login: osma
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-26T09:28:52Z
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-26T10:03:34Z
user: login: github-advanced-security[bot]
state: COMMENTED
author_association: NONE
submitted_at: 2025-03-27T12:57:51Z
user: login: juhoinkinen
state: COMMENTED
author_association: MEMBER
submitted_at: 2025-03-27T14:09:45Z
user: login: juhoinkinen
body: Yes, also I  think this makes the vocabulary and subject index functionality better. :+1: 

There were some tests [here](https://github.com/NatLibFi/Annif/pull/840/files#diff-57f229f7efa9436fe0378c6846208bbfeb55693c212a31648eec5a4ab3eef084) and [here](https://github.com/NatLibFi/Annif/pull/840/files#diff-34e190a882644e0eb5bbd4e9b7f4a640b406ebea839b3a539a20af4efc2cd79a) where variable renaming for the return value for `annif.vocab.VocabFile*` could be considered as those lines are now being edited (there is `subjects` and `corpus`). 
state: APPROVED
author_association: MEMBER
submitted_at: 2025-03-27T14:41:26Z

Review comments:
diff_hunk: @@ -17,6 +17,9 @@
 
     import numpy as np
 
+    from annif.vocab import SubjectIndex
path: annif/corpus/subject.py
user: login: github-advanced-security[bot]
body: ## Module-level cyclic import

'SubjectIndex' may not be defined if module [annif.vocab](1) is imported before module [annif.corpus.subject](2), as the [definition](3) of SubjectIndex occurs after the cyclic [import](4) of annif.corpus.subject.

[Show more details](https://github.com/NatLibFi/Annif/security/code-scanning/78)
created_at: 2025-03-26T08:21:29Z
author_association: NONE
diff_hunk: @@ -15,7 +16,7 @@
     from rdflib.graph import Graph
 
     from annif.corpus.skos import SubjectFileSKOS
-    from annif.corpus.subject import SubjectCorpus, SubjectIndex
+    from annif.corpus.subject import Subject, SubjectCorpus
path: annif/vocab.py
user: login: github-advanced-security[bot]
body: ## Cyclic import

Import of module [annif.corpus.subject](1) begins an import cycle.

[Show more details](https://github.com/NatLibFi/Annif/security/code-scanning/77)
created_at: 2025-03-26T08:21:29Z
author_association: NONE
diff_hunk: @@ -17,6 +17,9 @@
 
     import numpy as np
 
+    from annif.vocab import SubjectIndex
path: annif/corpus/subject.py
user: login: osma
body: This is a bad suggestion on several levels:

1. The supposed cyclic import (subject.py line 20) is within an `if TYPE_CHECKING:` block, so won't actually be evaluated at runtime.
2. The proposed fixes for the two `subjects()` methods (lines 50 and 101) are useless because those methods don't actually need the `SubjectIndex` type at all.

The only place `SubjectIndex` is mentioned in `subjects.py`, apart from the conditional import, is on line 150 where it's used as a type hint for the `from_string` method.
created_at: 2025-03-26T09:28:51Z
author_association: MEMBER
diff_hunk: @@ -17,8 +17,7 @@
 
     import numpy as np
 
-logger = annif.logger.getChild("subject")
-logger.addFilter(annif.util.DuplicateFilter())
+    from annif.vocab import SubjectIndex
path: annif/corpus/subject.py
user: login: github-advanced-security[bot]
body: ## Module-level cyclic import

'SubjectIndex' may not be defined if module [annif.vocab](1) is imported before module [annif.corpus.subject](2), as the [definition](3) of SubjectIndex occurs after the cyclic [import](4) of annif.corpus.subject.

[Show more details](https://github.com/NatLibFi/Annif/security/code-scanning/80)
created_at: 2025-03-26T10:03:34Z
author_association: NONE
diff_hunk: @@ -15,10 +16,11 @@
     from rdflib.graph import Graph
 
     from annif.corpus.skos import SubjectFileSKOS
-    from annif.corpus.subject import SubjectCorpus, SubjectIndex
+    from annif.corpus.subject import Subject, SubjectCorpus
path: annif/vocab.py
user: login: github-advanced-security[bot]
body: ## Cyclic import

Import of module [annif.corpus.subject](1) begins an import cycle.

[Show more details](https://github.com/NatLibFi/Annif/security/code-scanning/79)
created_at: 2025-03-26T10:03:34Z
author_association: NONE
diff_hunk: @@ -180,3 +177,45 @@
     assert len(concepts) == 2
     assert "http://example.org/dummy" in concepts
     assert "http://example.org/none" in concepts
+
+
+def test_subject_by_uri(subject_index):
+    subj_id = subject_index.by_uri("http://www.yso.fi/onto/yso/p7141")
+    assert subject_index[subj_id].labels["fi"] == "sinetit"
+
+
+def test_subject_by_uri_missing(subject_index):
+    subj_id = subject_index.by_uri("http://nonexistent")
+    assert subj_id is None
+
+
+def test_subject_by_label(subject_index):
+    subj_id = subject_index.by_label("sinetit", "fi")
+    assert subject_index[subj_id].uri == "http://www.yso.fi/onto/yso/p7141"
+
+
+def test_subject_by_label_missing(subject_index):
+    subj_id = subject_index.by_label("nonexistent", "fi")
+    assert subj_id is None
+
+
+def test_subject_index_filter(subject_index):
+    subject_filter = annif.vocab.SubjectIndexFilter(
+        subject_index, exclude=["http://www.yso.fi/onto/yso/p7141"]
+    )
+
+    assert len(subject_index) == len(subject_filter)
+
+    assert subject_index.languages == subject_filter.languages
+
+    subj_id = subject_index.by_uri("http://www.yso.fi/onto/yso/p7141")
+    with pytest.raises(IndexError):
+        subject_filter[subj_id]
path: tests/test_vocab.py
user: login: github-advanced-security[bot]
body: ## Statement has no effect

This statement has no effect.

[Show more details](https://github.com/NatLibFi/Annif/security/code-scanning/83)
created_at: 2025-03-27T12:57:51Z
author_association: NONE
diff_hunk: @@ -157,13 +157,13 @@ def run_load_vocab(vocab_id, language, force, subjectfile):
     Load a vocabulary from a subject file.
     """
     vocab = cli_util.get_vocab(vocab_id)
-    if annif.corpus.SubjectFileSKOS.is_rdf_file(subjectfile):
+    if annif.vocab.VocabFileSKOS.is_rdf_file(subjectfile):
path: annif/cli.py
user: login: juhoinkinen
body: Would `vocab_file_path` or something like that be a better name for `subjectfile`? 
created_at: 2025-03-27T14:09:45Z
author_association: MEMBER

PR comments:
user: login: codecov[bot]
created_at: 2025-03-26T08:23:14Z
body: ## [Codecov](https://app.codecov.io/gh/NatLibFi/Annif/pull/840?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) Report
Attention: Patch coverage is `99.75845%` with `1 line` in your changes missing coverage. Please review.
> Project coverage is 99.63%. Comparing base [(`ee2f456`)](https://app.codecov.io/gh/NatLibFi/Annif/commit/ee2f4562294c89e601451bf6b77c019ea84cb84b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) to head [(`4784c7b`)](https://app.codecov.io/gh/NatLibFi/Annif/commit/4784c7b1fd720018278f1d6b63a4656cd4fde524?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi).
> Report is 17 commits behind head on main.

| [Files with missing lines](https://app.codecov.io/gh/NatLibFi/Annif/pull/840?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) | Patch % | Lines |
|---|---|---|
| [annif/project.py](https://app.codecov.io/gh/NatLibFi/Annif/pull/840?src=pr&el=tree&filepath=annif%2Fproject.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi#diff-YW5uaWYvcHJvamVjdC5weQ==) | 94.44% | [1 Missing :warning: ](https://app.codecov.io/gh/NatLibFi/Annif/pull/840?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi) |

<details><summary>Additional details and impacted files</summary>


```diff
@@            Coverage Diff             @@
##             main     #840      +/-   ##
==========================================
- Coverage   99.64%   99.63%   -0.01%     
==========================================
  Files          96       99       +3     
  Lines        7242     7350     +108     
==========================================
+ Hits         7216     7323     +107     
- Misses         26       27       +1     
```

</details>

[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/NatLibFi/Annif/pull/840?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi).   
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=NatLibFi).

<details><summary> :rocket: New features to boost your workflow: </summary>

- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
</details>
author_association: NONE
user: login: sonarqubecloud[bot]
created_at: 2025-03-28T15:36:48Z
body: ## [![Quality Gate Passed](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/checks/QualityGateBadge/qg-passed-20px.png 'Quality Gate Passed')](https://sonarcloud.io/dashboard?id=NatLibFi_Annif&pullRequest=840) **Quality Gate passed**  
Issues  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [1 New issue](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=840&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/accepted-16px.png '') [0 Accepted issues](https://sonarcloud.io/project/issues?id=NatLibFi_Annif&pullRequest=840&issueStatuses=ACCEPTED)

Measures  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0 Security Hotspots](https://sonarcloud.io/project/security_hotspots?id=NatLibFi_Annif&pullRequest=840&issueStatuses=OPEN,CONFIRMED&sinceLeakPeriod=true)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Coverage on New Code](https://sonarcloud.io/component_measures?id=NatLibFi_Annif&pullRequest=840&metric=new_coverage&view=list)  
![](https://sonarsource.github.io/sonarcloud-github-static-resources/v2/common/passed-16px.png '') [0.0% Duplication on New Code](https://sonarcloud.io/component_measures?id=NatLibFi_Annif&pullRequest=840&metric=new_duplicated_lines_density&view=list)  
  
[See analysis details on SonarQube Cloud](https://sonarcloud.io/dashboard?id=NatLibFi_Annif&pullRequest=840)


author_association: NONE
user: login: osma
created_at: 2025-03-28T15:41:02Z
body: I tested the exclusion support using TFIDF, Omikuji and MLLM backends. It seems to work as expected for both training and suggestion; the excluded subjects are simply ignored.

For TFIDF and Omikuji (associative methods), if a concept is excluded during train time, the model will not learn it at all. Even if I drop the exclude setting from the project configuration after training, the project will never suggest it.

For MLLM (lexical method), the model will not learn individual concepts, so if I train a project with a concept excluded, then drop the exclude setting, the model will be able to suggest it.
author_association: MEMBER