# Results of the Ontology Alignment Evaluation Initiative 2023

Mina Abd Nikooie Pour, Alsayed Algergawy, Patrice Buche, Leyla Jael Castro, Jiaoyan Chen, Adrien Coulet, Julien Cufi, Hang Dong, Omaima Fallatah, Daniel Faria, et al.

<figure><img src="image_1.png" /></figure>

## To cite this version:

Mina Abd Nikooie Pour, Alsayed Algergawy, Patrice Buche, Leyla Jael Castro, Jiaoyan Chen, et al. Results of the Ontology Alignment Evaluation Initiative 2023. 18. International Workshop on Ontology Matching (OM 2023), Nov 2023, Athènes, Greece. pp.97-139. [hal-04366893](https://hal.inria.fr/04366893v1)

HAL Id: hal-04366893

https://inria.hal.science/hal-04366893v1

Submitted on 29 Dec 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
# Results of the Ontology Alignment Evaluation Initiative 2023

Mina Abd Nikooie Pour<fnref n="1" />, Alsayed Algergawy<fnref n="2" />,<fnref n="3" />, Patrice Buche<fnref n="4" />, Leyla J. Castro<fnref n="5" />, Jiaoyan Chen<fnref n="6" />, Adrien Coulet<fnref n="7" />,<fnref n="8" />, Julien Cufi<fnref n="4" />, Hang Dong<fnref n="9" />, Omaima Fallatah<fnref n="10" />, Daniel Faria<fnref n="11" />, Irini Fundulaki<fnref n="12" />, Sven Hertling<fnref n="13" />, Yuan He<fnref n="9" />, Ian Horrocks<fnref n="9" />, Martin Huschka<fnref n="14" />, Liliana Ibanescu<fnref n="15" />, Sarika Jain<fnref n="16" />, Ernesto Jiménez-Ruiz<fnref n="17" />, Naouel Karam<fnref n="18" />, Patrick Lambrix<fnref n="1" />, Huanyu Li<fnref n="1" />, Ying Li<fnref n="1" />, Pierre Monnin<fnref n="19" />, Engy Nasr<fnref n="20" />, Heiko Paulheim<fnref n="13" />, Catia Pesquita<fnref n="21" />, Tzanina Saveta<fnref n="12" />, Pavel Shvaiko<fnref n="22" />, Guilherme Sousa<fnref n="23" />, Cassia Trojahn<fnref n="23" />, Jana Vatascinova<fnref n="24" />, Mingfang Wu<fnref n="25" />, Beyza Yaman<fnref n="26" />, Ondrej Zamazal<fnref n="24" /> and Lu Zhou<fnref n="27" />

## Abstract

The Ontology Alignment Evaluation Initiative (OAEI) aims at comparing ontology matching systems on precisely defined test cases. These test cases can be based on ontologies of different levels of complexity and use different evaluation modalities. The OAEI 2023 campaign offered 15 tracks and was attended by 16 participants. This paper is an overall presentation of that campaign.

<footnote n="1">¹Linköping University & Swedish e-Science Research Centre, Linköping, Sweden</footnote>

<footnote n="2">²Heinz Nixdorf Chair for Distributed Information Systems, Friedrich Schiller University Jena, Germany</footnote>

<footnote n="3">³Chair of Data and Knowledge Engineering, University of Passau, Germany</footnote>

<footnote n="4">⁴UMR IATE, INRAE, University of Montpellier, France</footnote>

<footnote n="5">⁵ZB MED Information Centre for Life Sciences, Germany</footnote>

<footnote n="6">⁶Department of Computer Science, The University of Manchester, UK</footnote>

<footnote n="7">⁷Inria Paris, France</footnote>

<footnote n="8">⁸Centre de Recherche des Cordeliers, Inserm, Université Paris Cité, Sorbonne Université, France</footnote>

<footnote n="9">⁹Department of Computer Science, University of Oxford, UK</footnote>

<footnote n="10">¹⁰Department of Data Science, Umm Al-Qura University, Saudi Arabia</footnote>

<footnote n="11">¹¹INESC-ID / IST, University of Lisbon, Portugal</footnote>

<footnote n="12">¹²Institute of Computer Science-FORTH, Heraklion, Greece</footnote>

<footnote n="13">¹³Data and Web Science Group, University of Mannheim, Germany</footnote>

<footnote n="14">¹⁴Fraunhofer Institute for High-Speed Dynamics, Ernst-Mach-Institut, EMI, Germany</footnote>

<footnote n="15">¹⁵Université Paris-Saclay, INRAE, AgroParisTech, UMR MIA Paris-Saclay, France</footnote>

<footnote n="16">¹⁶National Institute of Technology Kurukshetra, India</footnote>

<footnote n="17">¹⁷City, University of London, UK & SIRIUS, University of Oslo, Norway</footnote>

<footnote n="18">¹⁸Institute for Applied Informatics, University of Leipzig, Germany</footnote>

<footnote n="19">¹⁹University Côte d'Azur, Inria, CNRS, I3S, France</footnote>

<footnote n="20">²⁰Albert Ludwig University of Freiburg, Germany</footnote>

<footnote n="21">²¹LASIGE, Faculdade de Ciências, Universidade de Lisboa, Portugal</footnote>

<footnote n="22">²²Trentino Digitale SpA, Trento, Italy</footnote>

<footnote n="23">²³Institut de Recherche en Informatique de Toulouse, France</footnote>

<footnote n="24">²⁴Prague University of Economics and Business, Czech Republic</footnote>

<footnote n="25">²⁵Australian Research Data Commons</footnote>

<footnote n="26">²⁶ADAPT Centre, Trinity College Dublin</footnote>

<footnote n="27">²⁷Flatfee Corp, USA</footnote>
# 1. Introduction

The Ontology Alignment Evaluation Initiative¹ (OAEI) is a coordinated international initiative, which organizes the evaluation of ontology matching systems [1, 2], and which has been run for eighteen years now. The main goal of the OAEI is to compare systems and algorithms openly and on the same basis to allow anyone to conclude the best ontology matching strategies. Furthermore, the ambition is that, from such evaluations, developers can improve their systems and offer better tools addressing the evolving application needs.

Two first events were organized in 2004: (i) the Information Interpretation and Integration Conference (I3CON) held at the NIST Performance Metrics for Intelligent Systems (PerMIS) workshop and (ii) the Ontology Alignment Contest held at the Evaluation of Ontology-based Tools (EON) workshop of the annual International Semantic Web Conference (ISWC) [3]. Then, a unique OAEI campaign occurred in 2005 at the workshop on Integrating Ontologies held in conjunction with the International Conference on Knowledge Capture (K-Cap) [4]. From 2006 until the present, the OAEI campaigns were held at the Ontology Matching workshop, co-located with ISWC [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21], which this year took place in Athens, Greece².

Since 2011, we have been using an environment for automatically processing evaluations which was developed within the SEALS (Semantic Evaluation At Large Scale) project³. SEALS provided a software infrastructure for automatically executing evaluations and evaluation campaigns for typical semantic web tools, including ontology matching. Since OAEI 2017, a novel evaluation environment called HOBBIT (Section 2.1) was adopted for the HOBBIT Link Discovery track, and later extended to enable the evaluation of other tracks. Some tracks are run exclusively through SEALS and others through HOBBIT, but several allow participants to choose their preferred platform. Since last year, the MELT framework [22] has been adopted to facilitate the SEALS and HOBBIT wrapping and evaluation. This year, most tracks have adopted MELT as their evaluation platform.

This paper synthesizes the 2023 evaluation campaign and introduces the results provided in the participants’ papers. The remainder of the paper is organized as follows: in Section 2, we present the overall evaluation methodology; in Section 3, we present the tracks and datasets; in Section 4 we present and discuss the results; and finally, Section 5 discusses the lessons learned.

OM 2023: The 18th International Workshop on Ontology Matching collocated with the 22nd International Semantic Web Conference ISWC-2023 November 7th, 2023, Athens, Greece

<figure><img src="image_3.png" /></figure>

© 2023 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).

<figure><img src="image_4.png" /></figure>

CEUR Workshop Proceedings (CEUR-WS.org)

¹http://oaei.ontologymatching.org

²http://om2023.ontologymatching.org

³http://www.seals-project.eu
## 2. Methodology

### 2.1. Evaluation platforms

The OAEI evaluation was conducted in one of three alternative platforms: the SEALS client, the HOBBIT platform, or the MELT framework. All of them have the goal of ensuring reproducibility and comparability of the results across matching systems. As of this campaign, the use of the SEALS client and packaging format is deprecated in favor of MELT, with the sole exception of the Interactive Matching track, as simulated interactive matching is not yet supported by MELT.

The **SEALS client** was developed in 2011. It is a Java-based command line interface for ontology matching evaluation, which requires system developers to implement an interface and to wrap their tools in a predefined way, including all required libraries and resources.

The **HOBBIT platform**<fnref n="4" /> was introduced in 2017. It is a web interface for linked data and ontology matching evaluation, which requires systems to be wrapped inside docker containers and includes a SystemAdapter class, then being uploaded into the HOBBIT platform [23].

The **MELT framework**<fnref n="5" /> [22] was introduced in 2019 and is under active development. It allows the development, evaluation, and packaging of matching systems for evaluation interfaces like SEALS or HOBBIT. It further enables developers to use Python or any other programming language in their matching systems, which beforehand had been a hurdle for OAEI participants. The evaluation client<fnref n="6" /> allows organizers to evaluate packaged systems whereby multiple submission formats are supported (SEALS packages or matchers implemented as Web services). Starting with this year, the MELT framework also supports the SSSOM [24] format. Therefore, systems producing an alignment in the SSSOM format can be evaluated as well.

All platforms compute the standard evaluation metrics against the reference alignments: precision, recall, and F-measure. In test cases requiring different evaluation modalities, evaluation was carried out *a posteriori*, using the alignments produced by the matching systems.

### 2.2. Submission formats

This year, three submission formats were allowed: (1) SEALS package, (2) HOBBIT, and (3) MELT Web interface. With the increasing usage of other programming languages than Java and increasing hardware requirements for matching systems, since 2021 the MELT Web interface was introduced to address this issue. It mainly consists of a technology-independent HTTP interface<fnref n="7" /> which participants can implement as they wish. Alternatively, they can use the MELT framework to assist them, as it can be used to wrap any matching system as docker container implementing the HTTP interface. In 2023, 12 systems were submitted as MELT Web docker container, 3 systems were submitted as SEALS package, 1 system was uploaded to the HOBBIT platform, and one system implemented the Web interface directly and provided hosting for the system.

In this year we also allowed to submit alignment files in addition to the executable system in case it requires substantial hardware or software resources.

<footnote n="4">⁴https://project-hobbit.eu/outcomes/hobbit-platform/</footnote>

<footnote n="5">⁵https://github.com/dwslab/melt</footnote>

<footnote n="6">⁶https://dwslab.github.io/melt/matcher-evaluation/client</footnote>

<footnote n="7">⁷https://dwslab.github.io/melt/matcher-packaging/web</footnote>
## 2.3. OAEI campaign phases

As in previous years, the OAEI 2023 campaign was divided into three phases: preparatory, execution, and evaluation.

In the **preparatory phase**, the test cases were provided to participants in an initial assessment period between June 30^th^ and July 31^st^, 2023. The goal of this phase is to ensure that the test cases make sense to participants, and give them the opportunity to provide feedback to organizers on the test case as well as potentially report errors. At the end of this phase, the final test base was frozen and released.

During the ensuing **execution phase**, participants test and potentially develop their matching systems to automatically match the test cases. Participants can self-evaluate their results either by comparing their output with the reference alignments or by using either of the evaluation platforms. They can tune their systems with respect to the non-blind evaluation as long as they respect the rules of the OAEI. Participants were required to register their systems by July 31^st^ and make a preliminary evaluation by August 31^st^. The execution phase was terminated on September 30^th^, 2023, at which date participants had to submit the (near) final versions of their systems (SEALS-wrapped and/or HOBBIT-wrapped).

During the **evaluation phase**, systems were evaluated by all track organizers. In case minor problems were found during the initial stages of this phase, they were reported to the developers, who were given the opportunity to fix and resubmit their systems. Initial results were provided directly to the participants, whereas final results for most tracks were published on the respective OAEI web pages before the workshop.

# 3. Tracks and Test Cases

This year's OAEI campaign consisted of 15 tracks, all of them including OWL ontologies while only one also including SKOS thesauri, namely the Biodiversity and the Ecology track. They can be grouped into:

- Schema matching tracks, which have as objective matching ontology classes and/or properties.

- Instance matching tracks, which have as objective matching ontology instances.

- Instance and schema matching tracks, which involve both of the above.

- Complex matching tracks, which have as objective finding complex correspondences between ontology entities.

- Interactive tracks, which simulate user interaction to enable the benchmarking of interactive matching algorithms.

The tracks are summarized in Table 1 and detailed in the following sections.
<table><caption>**Table 1**
Tracks in OAEI 2023.</caption><thead><tr><th>test</th><th>formalism</th><th>relations</th><th>confidence</th><th>modalities</th><th>language</th><th>SEALS</th><th>HOBBIT</th><th>MELT</th></tr></thead><tbody><tr><td colspan="9"><strong>T-Box/Schema matching</strong></td></tr><tr><td>anatomy</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td>✓</td><td></td><td>✓</td></tr><tr><td>conference</td><td>OWL</td><td>=, &lt;=</td><td>[0 1]</td><td>open+blind</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>multifarm</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open+blind</td><td>AR, CZ,<br>CN, DE,<br>EN, ES,<br>FR, IT,<br>NL, RU, PT</td><td></td><td></td><td>✓</td></tr><tr><td>complex</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open+blind</td><td>EN, ES</td><td></td><td></td><td>✓</td></tr><tr><td>food</td><td>OWL</td><td>=, &lt;=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>interactive</td><td>OWL</td><td>=, &lt;=</td><td>[0 1]</td><td>open</td><td>EN</td><td>✓</td><td></td><td></td></tr><tr><td>bio-ML</td><td>OWL</td><td>=, &lt;=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>biodiv</td><td>OWL/SKOS</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>mse</td><td>OWL</td><td>=, &lt;=, &gt;=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>crosswalks</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td>common knowl. graph</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td colspan="9"><strong>Instance and schema matching</strong></td></tr><tr><td>knowledge graph</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr><tr><td colspan="9"><strong>Instance matching or link discovery</strong></td></tr><tr><td>spimbench</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open+blind</td><td>EN</td><td>✓</td><td></td><td></td></tr><tr><td>link discovery</td><td>OWL</td><td>=</td><td>[0 1]</td><td>open</td><td>EN</td><td>✓</td><td></td><td></td></tr><tr><td>pharmacogenomics</td><td>OWL</td><td>=, &lt;, &gt;, Close, Related</td><td>[0 1]</td><td>open</td><td>EN</td><td></td><td></td><td>✓</td></tr></tbody></table>

## 3.1. Anatomy

The anatomy track comprises a single test case consisting of matching two fragments of biomedical ontologies which describe the human anatomy<fnref n="8" /> (3304 classes) and the anatomy of the mouse<fnref n="9" /> (2744 classes). The evaluation is based on a manually curated reference alignment. This dataset has been used since 2007 with some improvements over the years [25].

Systems are evaluated with the standard parameters of precision, recall, F-measure. Additionally, recall+ is computed by excluding trivial correspondences (i.e., correspondences that have the same normalized label). Alignments are also checked for coherence using the Pellet reasoner. The evaluation was carried out on a machine with a 5 core CPU @ 1.80 GHz with 16GB allocated RAM, using the MELT framework. For some systems, the SEALS client has been used. However, the evaluation parameters were computed *a posteriori*, after removing from the alignments produced by the systems, correspondences expressing relations other than equivalence, as well as trivial correspondences in the oboInOwl namespace (e.g., oboInOwl#Synonym = oboInOwl#Synonym). The results obtained with the SEALS client vary in some cases by 0.5% compared to the results presented in Section 4.2.

## 3.2. Conference

The conference track features two test cases. The main test case is a suite of 21 matching tasks corresponding to the pairwise combination of 7 moderately expressive ontologies describing the

<footnote n="8">⁸https://www.cancer.gov/cancertopics/cancerlibrary/terminologyresources</footnote>

<footnote n="9">⁹http://www.informatics.jax.org/searches/AMA_form.shtml</footnote>
domain of organizing conferences. The dataset and its usage are described in [26]. This year we
again run a second test case consisting of a suite of three tasks of matching DBpedia ontology
(filtered to the dbpedia namespace) and three ontologies from the conference domain.

For the main test case the track uses several reference alignments for evaluation: the old (and
not fully complete) manually curated open reference alignment, *ra1*; an extended, also manually
curated version of this alignment, *ra2*; a version of the latter corrected to resolve violations of
conservativity, *rar2*; and an uncertain version of *ra1* produced through crowd-sourcing, where
the score of each correspondence is the fraction of people in the evaluation group that agree
with the correspondence. The latter reference was used in two evaluation modalities: *discrete*
and *continuous* evaluation. In the former, correspondences in the uncertain reference alignment
with a score of at least 0.5 are treated as correct whereas those with lower score are treated
as incorrect, and standard evaluation parameters are used to evaluated systems. In the latter,
weighted precision, recall and F-measure values are computed by taking into consideration the
actual scores of the uncertain reference, as well as the scores generated by the matching system.
For the sharp reference alignments (*ra1*, *ra2* and *rar2*), the evaluation is based on the standard
parameters, as well the F₀.₅-measure and F₂-measure and on conservativity and consistency
violations. Whereas F₁ is the harmonic mean of precision and recall where both receive equal
weight, F₂ gives higher weight to recall than precision and F₀.₅ gives higher weight to precision
higher than recall. The second test case contains open reference alignment and systems were
evaluated using the standard metrics.

Two baseline matchers are used to benchmark the systems: edna string edit distance matcher;
and StringEquiv string equivalence matcher as in the anatomy test case.

3.3. Multifarm

The multifarm track [27] aims at evaluating the ability of matching systems to deal with ontologies
in different natural languages. This dataset results from the translation of 7 ontologies from the
conference track (cmt, conference, confOf, iasted, sigkdd, ekaw and edas) into 10 languages:
Arabic (ar), Chinese (cn), Czech (cz), Dutch (nl), French (fr), German (de), Italian (it), Portuguese
(pt), Russian (ru), and Spanish (es). The dataset is composed of 55 pairs of languages, with 49
matching tasks for each of them, taking into account the alignment direction (e.g. cmt<sub>en</sub> → edas<sub>de</sub>
and cmt<sub>de</sub> → edas<sub>en</sub> are distinct matching tasks). While part of the dataset is openly available, all
matching tasks involving the *edas* and *ekaw* ontologies (resulting in 55 × 24 matching tasks) are
used for blind evaluation.

We consider two test cases: i) those tasks where two different ontologies (cmt→edas, for instance) have been translated into two different languages; and ii) those tasks where the same ontology (cmt→cmt) has been translated into two different languages. For the tasks of type ii), good results are not only related to the use of specific techniques for dealing with cross-lingual ontologies, but also on the ability to exploit the identical structure of the ontologies. This year, we report the results on different ontologies (i).

The reference alignments used in this track derive directly from the manually curated Confer-
ence *ra1* reference alignments. In 2021, alignments have been manually evaluated by domain
experts. The evaluation is blind. The systems have been executed on a Ubuntu Linux machine
configured with 32GB of RAM running under a Intel Core CPU 2.00GHz x8 cores. The evalua-
tion was performed using the MELT platform. Every participating system was executed in its
standard setting and we compare precision, recall and F-measure as well as the computation time.

**3.4. Complex Matching**

The complex matching track is meant to evaluate the matchers based on their ability to gen-
erate complex alignments. A complex alignment is composed of complex correspondences
typically involving more than two ontology entities, such as o₁:AcceptedPaper ≡ o₂:Paper ⊓
o₂:hasDecision.o₂:Acceptance.

This year the track run with two data sets from the conference domain: Conference and Populated Conference, as the other complex sub-tracks (**Hydrography**, **GeoLink**, **Populated GeoLink Populated Enslaved**, and **Taxon** datasets) have been discontinued.

The **Conference** dataset comprises three ontologies: cmt, conference, and ekaw from the conference dataset. The reference alignment was created as a consensus between experts. To allow matchers which rely on instances to participate over the Conference complex track, the **Populated Conference** data set is composed of 5 conference ontologies populated with more or less common instances, resulting in 6 datasets: (6 versions on the repository: v0, v20, v40, v60, v80 and v100). Details on the population and evaluation modalities are available<fnref n="10" />.

The systems have been executed on a Ubuntu Linux machine configured with 32GB of RAM running under a Intel Core CPU 2.00GHz x8 processors.

**3.5. Food**

The Food Nutritional Composition track aims at finding alignments between food concepts
from CIQUAL<fnref n="11" />, the French food nutritional composition database, and food concepts from
SIREN<fnref n="12" />, the Scientific Information and Retrieval Exchange Network of the US Food and Drug
administration. Foods from both databases are described in LanguaL<fnref n="13" />, a well-known multilingual
thesaurus using faceted classification. LanguaL stands for “Langua aLimentaria” or “language of
food”; more than 40,000 foods used in food composition databases are described using LanguaL.

In [28], a method to provide OWL modelling of food concepts from both datasets, CIQUAL<sup>14</sup>
and SIREN<sup>15</sup>, and a gold standard are presented.

The evaluation was performed using the MELT platform. Every participating system was
executed in its standard setting and we compare precision, recall and F-measure as well as the
computation time.

**3.6. Interactive Matching**

The interactive matching track aims to assess the performance of semi-automated matching
systems by simulating user interaction [29, 30, 31]. The evaluation thus focuses on how interaction

<footnote n="10">¹⁰https://framagit.org/IRIT_UT2J/conference-dataset-population</footnote>

<footnote n="11">¹¹https://ciqual.anses.fr/</footnote>

<footnote n="12">¹²http://langual.org/langual_indexed_dataset.asp</footnote>

<footnote n="13">¹³https://www.langual.org/default.asp</footnote>

<footnote n="14">¹⁴https://entrepot.recherche.data.gouv.fr/dataset.xhtml?persistentId=doi:10.15454/6CEYU3</footnote>

<footnote n="15">¹⁵https://entrepot.recherche.data.gouv.fr/dataset.xhtml?persistentId=doi:10.15454/5LLGVY</footnote>
with the user improves the matching results. Currently, this track does not evaluate the user
experience or the user interfaces of the systems [32, 30].

The interactive matching track is based on the datasets from the Anatomy and Conference tracks, which have been previously described. It relies on the SEALS client's *Oracle* class to simulate user interactions. An interactive matching system can present a collection of correspondences simultaneously to the oracle, telling the system whether that correspondence is correct or not. If a system presents up to three correspondences together and each correspondence presented has a mapped entity (i.e., class or property) in common with at least one other correspondence presented, the oracle counts this as a single interaction, under the rationale that this corresponds to a scenario where a user is asked to choose between conflicting candidate correspondences. To simulate the possibility of user errors, the oracle can be set to reply with a given error probability (randomly, from a uniform distribution). We evaluated systems with four different error rates: 0.0 (perfect user), 0.1, 0.2, and 0.3.

In addition to the standard evaluation parameters, we also compute the number of requests made by the system, the total number of distinct correspondences asked, the number of positive and negative answers from the oracle, the performance of the system according to the oracle (to assess the impact of the oracle errors on the system) and finally, the performance of the oracle itself (to assess how erroneous it was).

The evaluation was carried out on a server with 3.46 GHz (6 cores) and 8GB RAM allocated to the matching systems. For systems requiring more RAM, the evaluation was carried out on a computer with an AMD Ryzen 7 5700G 3.80 GHz CPU and 32GB RAM, with 10GB of max heap space allocated to java. Each system was run ten times and the final result of a system for each error rate represents the average of these runs. For the Conference dataset with the ra1 alignment, precision and recall correspond to the micro-average over all ontology pairs, whereas the number of interactions is the total number of interactions for all the pairs.

## 3.7. Bio-ML

The Bio-ML track [33] incorporates both *equivalence* and *subsumption* ontology matching (OM) tasks for biomedical ontologies, with ground truth (equivalence) mappings extracted from Mondo [34] and UMLS [35] (see Table 2). Mondo aims to integrate disease concepts worldwide, while UMLS is a meta-thesaurus for the biomedical domain. Based on techniques (ontology pruning, subsumption mapping construction, negative candidate mapping generation, etc.) proposed in [33], we introduced five OM pairs with their information reported in Table 3. Each OM pair is accompanied with both equivalence and subsumption matching tasks; each matching task has two data split settings, i.e., *unsupervised* setting with no training mappings, and *semi-supervised* setting with 30% ground truth mappings for training/validation. In the 2023 edition, we made several significant updates:

- *Logical module enrichment:* we adopted locality-based logical modules [36] to enrich the existing pruned ontologies to provide more contexts for alignment; the added entities are annotated as *"not used in alignment"* and will be ignored in evaluation.

- *Bio-LLM sub-track:* To support more efficient evaluation of large language model-based OM, we introduced a special sub-track called Bio-LLM [37], which consists of challenging
subsets of NCIT-DOID and SNOMED-FMA (Body) datasets, along with tailored evaluation metrics.

- Simplified file structure and task settings: We also re-organised the structure of the dataset files, and reduced the task settings such that the unsupervised and semi-supervised settings share the same testing set for ranking evaluation.

For evaluation, in [33] we proposed both *global matching* and *local ranking*; the former aims to evaluate the overall performance by computing Precision, Recall, and F1 metrics for the output mappings against the reference mappings, while the latter aims to evaluate the ability to distinguish the correct mapping out of several challenging negatives by ranking metrics Hits@K and MRR. Note that subsumption mappings are inherently incomplete, so only local ranking evaluation is applied for subsumption matching. For the special sub-track Bio-LLM introduced in the 2023 edition, both matching and ranking metrics are used but they are tailored to the subsets, along with an additional metric called rejection rate to examine if systems can reject all plausible mappings for entities that actually have no alignment.

<table><caption>**Table 2**
Information of the source ontologies used for creating the OM datasets in Bio-ML.</caption><thead><tr><th>Mapping Source</th><th>Ontology</th><th>Ontology Source & Version</th><th>#Classes</th></tr></thead><tbody><tr><td rowspan="4">Mondo</td><td>OMIM</td><td>Mondo<sup>16</sup></td><td>44,729</td></tr><tr><td>ORDO</td><td>BioPortal, V3.2</td><td>14,886</td></tr><tr><td>NCIT</td><td>BioPortal, V18.05d</td><td>140,144</td></tr><tr><td>DOID</td><td>BioPortal, 2017-11-28</td><td>12,498</td></tr><tr><td rowspan="3">UMLS</td><td>SNOMED</td><td>UMLS, US.2021.09.01<sup>17</sup></td><td>358,222</td></tr><tr><td>FMA</td><td>BioPortal, V4.14.0</td><td>104,523</td></tr><tr><td>NCIT</td><td>BioPortal, V21.02d</td><td>163,842</td></tr></tbody></table>

<table><caption>**Table 3**
Information of each OM dataset in Bio-ML, where the numbers of equivalence and subsumption reference mappings are reported in #Refs(≡) and #Refs (□), respectively.</caption><thead><tr><th>Mapping Source</th><th>Ontology Pair</th><th>Category</th><th>#Refs (≡)</th><th>#Refs (□)</th></tr></thead><tbody><tr><td rowspan="2">Mondo</td><td>OMIM-ORDO</td><td>Disease</td><td>3,721</td><td>103</td></tr><tr><td>NCIT-DOID</td><td>Disease</td><td>4,684</td><td>3,339</td></tr><tr><td rowspan="3">UMLS</td><td>SNOMED-FMA</td><td>Body</td><td>7,256</td><td>5,506</td></tr><tr><td>SNOMED-NCIT</td><td>Pharm</td><td>5,803</td><td>4,225</td></tr><tr><td>SNOMED-NCIT</td><td>Neoplas</td><td>3,804</td><td>213</td></tr></tbody></table>

We adopted a flexible way of evaluating participating systems. First, participants can freely choose any tasks and settings they would like to attend. Second, for systems that have been well-adapted to the MELT platform, we used MELT to produce the output mappings. Third, for systems that have been implemented elsewhere and are not easy to be made compatible with MELT, we used their source code. Fourth, we also allowed participants (with trust) to directly upload output mappings if their systems had not been published and had not been made compatible with MELT. In the final result tables, we used superscripts †, ‡, and * to indicate

<footnote n="16">¹⁶Created from OMIM texts by Mondo's pipeline tool available at: https://github.com/monarch-initiative/omim.</footnote>

<footnote n="17">¹⁷Created by the official snomed-owl-toolkit available at: https://github.com/IHTSDO/snomed-owl-toolkit.</footnote>
that the results came from MELT, source code implementation, and direct result submission, respectively. All our evaluations were conducted with the DeepOnto<fnref n="18" /> [38] library on a local machine with Intel Xeon Bronze 3204 CPU 1.90GHz x11 processors, 126GB RAM, and two Quadro RTX 8000 GPUs. The GPUs were mainly used for training systems that involve deep neural networks.

## 3.8. Biodiversity and Ecology

The biodiversity and ecology (biodiv) track is motivated by the GFBio<fnref n="19" /> (The German Federation for Biological Data) alongside its successor NFDI4Biodiversity<fnref n="20" /> and the AquaDiva<fnref n="21" /> projects, which aim at providing semantically enriched data management solutions for data capture, annotation, indexing and search [39, 40, 41]. In this track, we aim to motivate ontology matching systems to work on matching ontologies and thesauri used in the biodiversity and ecology domains, available via the BiodivPortal ontology repository<fnref n="22" />. For the current edition, we kept the matching task between the Environment Ontology (ENVO) and the Semantic Web for Earth and Environment Technology Ontology (SWEET) as these two ontologies have frequent updates.

In 2021, we added a task to align two biological taxonomies with rather different but complementary scopes: the well-known NCBI taxonomy (NCBITAXON), and TAXREF-LD [42]. No matching system was able to achieve this matching task due to the large size of the considered taxonomies. To cope with this issue since last year edition, we split the large matching task into a set of smaller, more manageable subtasks through the use of modularization [43]. We obtained six groups corresponding to the kingdoms: Animalia, Bacteria, Chromista, Fungi, Plantae and Protozoa, leading to six well balanced matching subtasks. In 2023, we partnered with the EcoPortal project<fnref n="23" /> to include two new matching tasks involving important thesauri in environmental sciences (originally developed in SKOS): finding alignments between the Macroalgae Traits Thesaurus (MACROALGAE) and the Macrozoobenthos Traits Thesaurus (MACROZOOBENTHOS) and between the Fish Traits Thesaurus (FISH) and the Zooplankton Traits Thesaurus (ZOOPLANKTON). Table 4 presents detailed information about the ontologies and thesauri used in this year's edition.

## 3.9. Material Sciences and Engineering (MSE)

Data in Material Sciences and Engineering (MSE) can be characterized by scarcity, complexity, and the presence of gaps. Therefore the MSE community aims for ontology-based data integration via decentralized data management architectures. Several actors using different ontologies result in the growing demand for automatic alignment of ontologies in the MSE domain.

The MSE track uses small to mid-sized ontologies common in the MSE field that are implemented with and without upper-level ontologies. The ontologies follow heterogeneous design

<footnote n="18">¹⁸https://krr-oxford.github.io/DeepOnto/#/</footnote>

<footnote n="19">¹⁹www.gfbio.org</footnote>

<footnote n="20">²⁰www.nfdi4biodiversity.org/en/</footnote>

<footnote n="21">²¹www.aquadiva.uni-jena.de</footnote>

<footnote n="22">²²biodivportal.gfbio.org/</footnote>

<footnote n="23">²³ecoportal.lifewatch.eu/</footnote>
<caption>**Table 4**</caption>

Biodiversity and Ecology track ontologies and thesauri.

<table><caption>**Table 5**</caption><thead><tr><th>Ontology/Thesaurus</th><th>Format</th><th>Version</th><th>Classes</th><th>Instances</th></tr></thead><tbody><tr><td>ENVO</td><td>OWL</td><td>2021-05-19</td><td>6,566</td><td>44</td></tr><tr><td>SWEET</td><td>OWL</td><td>2019-10-12</td><td>4,533</td><td>-</td></tr><tr><td>MACROALGAE</td><td>SKOS</td><td>2018-10-02</td><td>-</td><td>109</td></tr><tr><td>MACROZOOBENTHOS</td><td>SKOS</td><td>2023-07-11 (v1.1)</td><td>-</td><td>128</td></tr><tr><td>FISH</td><td>SKOS</td><td>2015-03-11</td><td>-</td><td>146</td></tr><tr><td>ZOOPLANKTON</td><td>SKOS</td><td>2019-05-27</td><td>-</td><td>57</td></tr><tr><td>NCBITAXON Animalia</td><td>OWL</td><td>2021-02-15</td><td>74729</td><td>-</td></tr><tr><td>TAXREF-LD Animalia</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>73528</td><td>-</td></tr><tr><td>NCBITAXON Bacteria</td><td>OWL</td><td>2021-02-15</td><td>326</td><td>-</td></tr><tr><td>TAXREF-LD Bacteria</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>312</td><td>-</td></tr><tr><td>NCBITAXON Chromista</td><td>OWL</td><td>2021-02-15</td><td>2344</td><td>-</td></tr><tr><td>TAXREF-LD Chromista</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>2290</td><td>-</td></tr><tr><td>NCBITAXON Fungi</td><td>OWL</td><td>2021-02-15</td><td>13149</td><td>-</td></tr><tr><td>TAXREF-LD Fungi</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>12732</td><td>-</td></tr><tr><td>NCBITAXON Plantae</td><td>OWL</td><td>2021-02-15</td><td>27013</td><td>-</td></tr><tr><td>TAXREF-LD Plantae</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>26302</td><td>-</td></tr><tr><td>NCBITAXON Protozoa</td><td>OWL</td><td>2021-02-15</td><td>538</td><td>-</td></tr><tr><td>TAXREF-LD Protozoa</td><td>OWL</td><td>2020-06-23 (v13.0)</td><td>501</td><td>-</td></tr></tbody></table>

The building blocks of the MSE track (MSE benchmark v1.1).

<table><thead><tr><th>Inputs</th><th>First Test Case</th><th>Second Test Case</th><th>Third Test Case</th></tr></thead><tbody><tr><td><strong>O1</strong></td><td>Reduced MaterialInformation</td><td>MaterialInformation</td><td>MaterialInformation</td></tr><tr><td><strong>O2</strong></td><td>MatOnto</td><td>MatOnto</td><td>EMMO</td></tr><tr><td><strong>R</strong></td><td>=, ⊂, ⊃ corresp.</td><td>= corresp.</td><td>= corresp.</td></tr><tr><td><strong>resources</strong></td><td colspan="3">Chemical Elements Dictionary (DICT), EMMO</td></tr></tbody></table>

principles with only partial overlap with each other. The current version v1.1²⁴ of the MSE track includes three test cases summarized in Table 5, where each test case consists of two MSE ontologies to be matched [O1; O2] as well as one manual reference alignment *R* that can be used for evaluation of the matching task. The benchmark also provides background knowledge *resources*.

The MSE track makes use of three different MSE ontologies in total, in each of which an ontology using an upper-level ontology is matched to one without an upper-level. The MaterialInformation[44] domain ontology was designed without upper-level ontology and serves as infrastructure for material information and knowledge exchange (545 classes, 98 properties, and 411 individuals). Three out of eight submodules of the MaterialInformation were merged to create the Reduced MaterialInformation (32 classes, 43 properties, and 17 individuals) for more efficient creation of the manual reference alignment in the First Test Case, see Table 5.

<footnote n="24">²⁴https://github.com/EngyNasr/MSE-Benchmark/releases/tag/v1.1</footnote>
The MatOnto Ontology v2.1<sup>25</sup> (847 classes, 96 properties and 131 individuals) bases on the upper-level ontology bfo2<sup>26</sup>. The Elementary Multiperspective Material Ontology (EMMO v1.0.0-alpha2)<sup>27</sup>, is a standard representational ontology framework based on current materials modeling and characterization knowledge incorporating an upper-, mid- and domain-level (451 classes, 35 properties). For every test case, a manual reference alignment <i>R</i> was created in close cooperation with MSE domain experts.

The evaluation was performed using the MELT platform on a Windows 11 system with Intel Core i7-1260P CPU @2.10GHz and 16 GB RAM. Every participating system was executed in its standard setting, and we compared precision, recall, and F-measure as well as the computation time. No background knowledge was used for evaluation.

## 3.10. Crosswalks Data Schema Matching

This track was introduced in 2022, aiming at evaluating the ability of systems to deal with the schema metadata matching task, in particular, with a collection of crosswalks from fifteen research data schemes to Schema.org [45, 46]. It is based on the work carried out by the Research Data Alliance (RDA) Research Metadata Schemas Working Group. The collection serves as a reference for data repositories when they develop their crosswalks, as well as an indication of semantic interoperability among the schemas.

The dataset is composed of 15 source research metadata describing datasets that have been aligned to Schema.org. The source schemas include discipline agnostic schemas Dublin Core, Data Catalogue Vocabulary (DCAT), Data Catalogue Vocabulary - Application Profile (DCAT-AP), Registry Interchange Format - Collections and Services (RIF-CS), DataCite Schema, Dataverse; and discipline schemas ISO19115-1, EOSC/EDMI, Data Tag Suite (DATS), Bioschemas, B2FIND, Data Documentation Initiative (DDI), European Clinical Research Infrastructure Network (ECRIN), Space Physics Archive Search and Extract (SPASE); as well as CodeMeta for software.

This year a subset of the 15 metadata schemas aligned to schema.org has been considered. This subset corresponds to the set of schemas and vocabularies for which an OWL/RDFS serialization is available. It involves Data Catalogue Vocabulary (DCAT-v3), Data Catalogue Vocabulary - Application Profile (DCAT-AP), DataCity, Dublin Core (DC), and ISO19115-1 schemas (ISO).

Using as a reference the manually established correspondences, the evaluation here will be based on the well-known measures of precision, recall, and F-measure. The systems have been executed on a Ubuntu Linux machine configured with 32GB of RAM running under an Intel Core CPU 2.00GHz x8 processors.

## 3.11. Common Knowledge Graphs

This track was introduced to OAEI in 2021, and it evaluates the ability of matching systems to match the schema (classes) in large cross-domain knowledge graphs such as DBpedia [47], YAGO [48] and NELL [49]. The dataset used for the evaluation is generated from DBpedia and

<footnote n="25"><sup>25</sup>https://raw.github.com/iNovexIrad/MatOnto-Ontologies/master/matonto-release.ttl</footnote>

<footnote n="26"><sup>26</sup>http://purl.obolibrary.org/obo/bfo/2.0/bfo.owl</footnote>

<footnote n="27"><sup>27</sup>https://raw.github.com/emmo-repo/EMMO/1.0.0-alpha2/emmo.owl</footnote>
<table><caption>**Table 6**
The number of classes and instances in the two common KGs benchmarks.</caption><thead><tr><td>Dataset</td><td>#Classes</td><td>#Instances</td></tr></thead><tbody><tr><td>DBpedia</td><td>138</td><td>631,461</td></tr><tr><td>NELL</td><td>134</td><td>1,184,377</td></tr><tr><td>YAGO</td><td>304</td><td>5,149,594</td></tr><tr><td>Wikidata</td><td>304</td><td>2,158,547</td></tr></tbody></table>

the Never-Ending Language Learner (NELL). While DBpedia is generated from structured data in Wikipedia's articles, NELL is an automatically generated knowledge graph with entities extracted from large-scale text corpus shared on websites. The automatic extraction process is one of the aspects that make common knowledge graphs different from ontologies, as they often result in less well-formatted and cross-domain datasets. In addition to the NELL and DBpedia test case, this year, we introduced a new test case for matching classes from YAGO and Wikidata [50]. The numbers of entities in the four KG datasets are illustrated in Table 6.

The NELL and DBpedia benchmark [51] was human-annotated and verified by experts. This gold standard is only a *partial gold standard* since not every class in each knowledge graph has an equivalent class in the opposite one. To avoid over-penalizing matches that may discover reasonable matches that are not included in the partial gold standard, our evaluation ignores any predicted matches where neither of the classes in that pair exists in a true positive pair with another class in the reference alignments. In terms of YAGO and Wikidata *gold standard*, it was originally created [52] and expanded according to OAEI standard as part of [50].

With respect to the reference alignment, matching systems were evaluated using standard precision, recall, and f-measure. The evaluation was carried out on a Linux virtual machine with 128 GB of RAM and 16 vCPUs (2.4 GHz) processors. The evaluation was performed using MELT for matchers wrapped using both SEALS, and the web packaging via Docker. As a baseline, we utilize a simple string matcher which is available through MELT.

## 3.12. Knowledge Graph

The Knowledge Graph track was run for the fourth year. The task of the track is to match pairs of knowledge graphs whose schema and instances have to be matched simultaneously. The individual knowledge graphs are created by running the DBpedia extraction framework on eight different Wikis from the Fandom Wiki hosting platform<fnref n="28" /> in the course of the DBkWik project [53, 54]. They cover different topics (movies, games, comics, and books) and three Knowledge Graph clusters sharing the same domain e.g., star trek, as shown in Table 7.

The evaluation is based on reference correspondences at both schema and instance levels. While the schema-level correspondences were created by experts, the instance correspondences were extracted from the wiki page itself. Due to the fact that not all interwiki links on a page represent the same concept, a few restrictions were made: 1) only links in sections with a header containing “link” are used, 2) all links are removed where the source page links to more than one concept in another wiki (ensures the alignments are functional), 3) multiple links which point to

<footnote n="28">²⁸https://www.wikia.com/</footnote>
<caption>**Table 7**</caption>

Characteristics of the Knowledge Graphs in the Knowledge Graph track and the sources they were created from.

<table><thead><tr><th>Source</th><th>Hub</th><th>Topic</th><th>#Instances</th><th>#Properties</th><th>#Classes</th></tr></thead><tbody><tr><td>Star Wars Wiki</td><td>Movies</td><td>Entertainment</td><td>145,033</td><td>700</td><td>269</td></tr><tr><td>The Old Republic Wiki</td><td>Games</td><td>Gaming</td><td>4,180</td><td>368</td><td>101</td></tr><tr><td>Star Wars Galaxies Wiki</td><td>Games</td><td>Gaming</td><td>9,634</td><td>148</td><td>67</td></tr><tr><td>Marvel Database</td><td>Comics</td><td>Comics</td><td>210,996</td><td>139</td><td>186</td></tr><tr><td>Marvel Cinematic Universe</td><td>Movies</td><td>Entertainment</td><td>17,187</td><td>147</td><td>55</td></tr><tr><td>Memory Alpha</td><td>TV</td><td>Entertainment</td><td>45,828</td><td>325</td><td>181</td></tr><tr><td>Star Trek Expanded Universe</td><td>TV</td><td>Entertainment</td><td>13,426</td><td>202</td><td>283</td></tr><tr><td>Memory Beta</td><td>Books</td><td>Entertainment</td><td>51,323</td><td>423</td><td>240</td></tr></tbody></table>

the same concept are also removed (ensures injectivity), 4) links to disambiguation pages were manually checked and corrected. Since we do not have a correspondence for each instance, class, and property in the graphs, this gold standard is only a *partial gold standard*.

The evaluation was executed on a virtual machine (VM) with 32GB of RAM and 16 vCPUs (2.4 GHz), with Debian 9 operating system and Openjdk version 1.8.0_265. For evaluating all possible submission formats, MELT framework is used. The corresponding code for evaluation can be found on Github<sup>29</sup>.

The alignments were evaluated based on precision, recall, and f-measure for classes, properties, and instances (each in isolation). The partial gold standard contained 1:1 correspondences, and we further assume that in each knowledge graph, only one representation of the concept exists. This means that if we have a correspondence in our gold standard, we count a correspondence to a different concept as a false positive. The count of false negatives is only increased if we have a 1:1 correspondence and it is not found by a matcher.

As a baseline, we employed two simple string-matching approaches. The source code for these matchers is publicly available<sup>30</sup>.

## 3.13. SPIMBENCH and Link Discovery

This year, only LogMap has participated in the SPIMBENCH and Link Discovery tracks. The organizers then made the decision not to run these tracks this year.

## 3.14. Pharmacogenomics

The Pharmacogenomics track is a new track proposed for OAEI 2023 that focuses on matching knowledge units from the pharmacogenomics domain. These units are *n*-ary tuples – so-called “pharmacogenomic relationships” – and involve drugs, genetic factors, and phenotypes (see Figure 1). A pharmacogenomic tuple states that patients being treated by the specified drugs while having the specified genetic factors may experience the given phenotypes.

In the Semantic Web formalisms, only binary predicates exist. That is why pharmacogenomic tuples are reified: tuples become individuals that are linked to their components with binary

<footnote n="29"><sup>29</sup>https://github.com/dwslab/melt/tree/master/examples/kgEvalCli</footnote>

<footnote n="30"><sup>30</sup>http://oaei.ontologymatching.org/2019/results/knowledgegraph/kgBaselineMatchers.zip</footnote>
predicates (Figure 1(c)). Hence, the task of matching pharmacogenomic tuples is [55]:

- An *instance matching task* that aims at finding alignments between individuals representing reified tuples;

- A *structure-based matching task* in which neighbors of reified tuples are compared to conclude the potential alignment between tuples. Recall that no other information exist about the tuples except their neighbors (*e.g.*, no labels).

To illustrate, two tuples involving the same drugs, genetic factors, and phenotypes will thus be connected to the same neighbors and are expected to be detected as identical.

Beside the arity of tuples, matchers need to face their potential incompleteness (*e.g.*, missing drugs) and heterogeneity (*e.g.*, a gene version like *CYP2C9*\*4 is more specific than the gene itself *CYP2C9*, the phenotype *hemorrhage* is more specific than the phenotype *vascular disorders*). *Different types of alignments* are thus expected to be identified between pharmacogenomic tuples, which is somehow unusual in an instance matching task. The Pharmacogenomics track features the identification of identical tuples (=), equivalent tuples (*Close*), tuples being more specific (<) or more general (>) than others, and tuples being related to some extent (*Related*). See [55, 56] for a detailed definition of these different alignment types between individuals.

To perform this alignment task, matchers can rely on additional background knowledge about components of pharmacogenomic tuples. This knowledge includes ontology classes instanciated by components of tuples (*i.e.* drugs, genetic factors, phenotypes) and their hierarchical organization, *partOf* links between gene versions and genes, *sameAs* links between identical drugs, genes, or phenotypes, and *dependsOn* links between complex phenotypes and their components (*e.g.*, “warfarin-induced bleeding” depends on “warfarin” and on “bleeding”).

To evaluate matchers and their scalability, the Pharmacogenomics track comprises three tasks involving respectively 10, 50, and 100% of the 50,435 pharmacogenomic tuples represented within the PGxLOD knowledge graph<fnref n="31" /> [57]. For each task, the selected pharmacogenomic tuples are evenly split into two ontologies to match. To take into account the specificity of the different alignment types that are expected, matchers are evaluated through two settings:

**Fine-grained setting** Only alignments of the exact type expected in the reference are considered correct. To illustrate, an output alignment (*e*₁, =, *e*₂) where (*e*₁, *Close*, *e*₂) was expected will be considered as incorrect. Precision, Recall, and F1-score are computed for each type of alignment.

**Coarse-grained setting** Any type of alignment between entities expected to be aligned will be considered as correct. To illustrate, an output alignment (*e*₁, =, *e*₂) where (*e*₁, *Close*, *e*₂) was expected will be considered as correct. Precision, Recall, and F1-score are computed globally accordingly.

<footnote n="31">³¹https://pgxlod.loria.fr/</footnote>
<figure><img src="image_5.png" /><figcaption>**Figure 1:** Graphical representation of an abstract (1(a)), an example (1(b)), and a reified (1(c)) pharmacogenomic relationships. The example relationship states that patients having the "*4" version of the *CYP2D6* gene will not experience the expected effect of codeine. gf stands for genetic factor, d for drug and p for phenotype.</figcaption></figure>

# 4. Results and Discussion

## 4.1. Participation

Following an initial period of growth, the number of OAEI participants has remained approximately constant since 2012, at slightly over 20. This year we count with 16 participating systems. Table 8 lists the participants and the tracks in which they competed. It is worth mentioning that the Bio-ML track has additional participants (e.g., BERTMap [58] and BERTSubs [59]) that are not listed in Table 8. This is because they need training and validation which are not yet fully supported by the OAEI evaluation platforms, and thus they were tested locally with Bio-ML results reported, but without an OAEI system submission. Some matching systems participated with different variants (Matcha and LogMap), whereas others were evaluated with different configurations, as requested by developers (see test case sections for details). The following sections summarize the results for each track.

## 4.2. Anatomy

The results for the Anatomy track are shown in Table 9. Of the 9 systems participating in the Anatomy track, 8 achieved an F-measure higher than the StringEquiv baseline. Two systems were first-time participants (SORBETMatcher, OLaLa). Long-term participating systems showed few changes in comparison with previous years with respect to alignment quality (precision, recall, F-measure, and recall+), size and run time. The exception were LogMapBio which increase in F-measure (from 0.895 to 0.898) and Matcha increased in recall+ (from 0.817 to 0.818), in size (from 1482 to 1484). In terms of run time, 5 out of 9 systems computed an alignment in less than 100 seconds. LogMapLt remains the system with the shortest runtime. Regarding quality, Matcha achieved the highest F-measure (0.941) and recall+ (0.818), but four other systems obtained an F-measure above 0.88 (OLaLa, SORBETMatcher, LogMapBio, and LogMap) which is at least as good as the best systems in OAEI 2007-2010. Like in previous years, there is no significant correlation between the quality of the generated alignment and the run time. Two systems produced coherent alignments (LogMapBio and LogMap).
<table><caption>**Table 8**

Participants and the status of their submissions.</caption><thead><tr><th>System</th><th>ALIN</th><th>AMD</th><th>GraphMatcher</th><th>LogMap</th><th>LogMapBio</th><th>LogMapLt</th><th>LogMapKG</th><th>LSMatch</th><th>LSMatch-Multilingual</th><th>Matcha</th><th>MatchaC</th><th>Matcha-DL</th><th>OLaLa</th><th>PropMatch</th><th>SORBETMatcher</th><th>TOMATO</th><th>Total=16</th></tr></thead><tbody><tr><td>anatomy</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td><td>●</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>9</td></tr><tr><td>conference</td><td>●</td><td>●</td><td>●</td><td>●</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td><td>●</td><td>11</td></tr><tr><td>multifarm</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>4</td></tr><tr><td>complex</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>3</td></tr><tr><td>food</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>4</td></tr><tr><td>interactive</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>2</td></tr><tr><td>bio-ML</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>●</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>7</td></tr><tr><td>biodiv</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>5</td></tr><tr><td>mse</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>4</td></tr><tr><td>common knowl. graph</td><td>○</td><td>●</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>●</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>7</td></tr><tr><td>crosswalks</td><td>○</td><td>○</td><td>○</td><td>●</td><td>●</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>5</td></tr><tr><td>knowledge graph</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>6</td></tr><tr><td>spimbench</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>1</td></tr><tr><td>link discovery</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>1</td></tr><tr><td>pharmacogenomics</td><td>○</td><td>○</td><td>○</td><td>●</td><td>○</td><td>●</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>●</td><td>○</td><td>○</td><td>○</td><td>○</td><td>○</td><td>5</td></tr></tbody><tfoot><tr><td>total</td><td>3</td><td>5</td><td>1</td><td>15</td><td>4</td><td>12</td><td>4</td><td>4</td><td>1</td><td>10</td><td>1</td><td>1</td><td>7</td><td>1</td><td>4</td><td>1</td><td></td><td></td><td></td></tr></tfoot></table>

<table><caption>**Table 9**

Anatomy results, ordered by F-measure. Runtime is measured in seconds; "size" is the number of correspondences in the generated alignment.</caption><thead><tr><th>System</th><th>Runtime</th><th>Size</th><th>Precision</th><th>F-measure</th><th>Recall</th><th>Recall+</th><th>Coherent</th></tr></thead><tbody><tr><td>Matcha</td><td>54</td><td>1484</td><td>0.951</td><td>0.941</td><td>0.931</td><td>0.818</td><td>-</td></tr><tr><td>OLaLa</td><td>9259</td><td>1470</td><td>0.924</td><td>0.91</td><td>0.896</td><td>0.726</td><td>-</td></tr><tr><td>SORBETMatcher</td><td>4032</td><td>1470</td><td>0.923</td><td>0.909</td><td>0.895</td><td>0.724</td><td>-</td></tr><tr><td>LogMapBio</td><td>1945</td><td>1578</td><td>0.88</td><td>0.898</td><td>0.916</td><td>0.778</td><td>✓</td></tr><tr><td>LogMap</td><td>10</td><td>1402</td><td>0.917</td><td>0.881</td><td>0.848</td><td>0.602</td><td>✓</td></tr><tr><td>AMD</td><td>36</td><td>1282</td><td>0.938</td><td>0.86</td><td>0.794</td><td>0.461</td><td>-</td></tr><tr><td>ALIN</td><td>374</td><td>1159</td><td>0.984</td><td>0.852</td><td>0.752</td><td>0.501</td><td>-</td></tr><tr><td>LogMapLt</td><td>3</td><td>1147</td><td>0.962</td><td>0.828</td><td>0.728</td><td>0.288</td><td>-</td></tr><tr><td>StringEquiv</td><td>-</td><td>946</td><td>0.997</td><td>0.766</td><td>0.622</td><td>0.000</td><td>-</td></tr><tr><td>LSMatch</td><td>16</td><td>1009</td><td>0.952</td><td>0.761</td><td>0.634</td><td>0.037</td><td>-</td></tr></tbody></table>

### 4.3. Conference

The conference evaluation results using the sharp reference alignment *rar2* are shown in Table 10. For the sake of brevity, only results with this reference alignment and considering both classes
and properties are shown. For more detailed evaluation results, please check the conference track's web page.

<caption>**Table 10**</caption>

The highest average $F_{[0.5|1|2]}$-measure and their corresponding precision and recall for each matcher with its $F_1$-optimal threshold (ordered by $F_1$-measure). Inc Align. means the number of incoherent alignments. Conser.V. means the total number of all conservative principle violations. Consist.V. means the total number of all consistency principle violations.

<table><thead><tr><th>System</th><th>Prec.</th><th>F<sub>0.5</sub>-m.</th><th>F<sub>1</sub>-m.</th><th>F<sub>2</sub>-m.</th><th>Rec.</th><th>Inc Align.</th><th>Conser.V.</th><th>Consist.V.</th></tr></thead><tbody><tr><td>GraphMatcher</td><td>0.71</td><td>0.72</td><td>0.74</td><td>0.76</td><td>0.77</td><td>8</td><td>172</td><td>85</td></tr><tr><td>SORBETMatcher</td><td>0.73</td><td>0.7</td><td>0.66</td><td>0.63</td><td>0.61</td><td>7</td><td>43</td><td>73</td></tr><tr><td>LogMap</td><td>0.76</td><td>0.71</td><td>0.64</td><td>0.59</td><td>0.56</td><td>0</td><td>2</td><td>0</td></tr><tr><td>Matcha</td><td>0.62</td><td>0.62</td><td>0.62</td><td>0.62</td><td>0.62</td><td>7</td><td>90</td><td>81</td></tr><tr><td>OLaLa</td><td>0.59</td><td>0.59</td><td>0.6</td><td>0.61</td><td>0.61</td><td>11</td><td>199</td><td>184</td></tr><tr><td>ALIN</td><td>0.82</td><td>0.7</td><td>0.57</td><td>0.48</td><td>0.44</td><td>0</td><td>2</td><td>0</td></tr><tr><td>edna</td><td>0.74</td><td>0.66</td><td>0.56</td><td>0.49</td><td>0.45</td><td></td><td></td><td></td></tr><tr><td>LogMapLt</td><td>0.68</td><td>0.62</td><td>0.56</td><td>0.5</td><td>0.47</td><td>0</td><td>21</td><td>0</td></tr><tr><td>AMD</td><td>0.82</td><td>0.68</td><td>0.55</td><td>0.46</td><td>0.41</td><td>1</td><td>2</td><td>6</td></tr><tr><td>LSMatch</td><td>0.83</td><td>0.69</td><td>0.55</td><td>0.46</td><td>0.41</td><td>3</td><td>97</td><td>18</td></tr><tr><td>StringEquiv</td><td>0.76</td><td>0.65</td><td>0.53</td><td>0.45</td><td>0.41</td><td></td><td></td><td></td></tr><tr><td>TOMATO</td><td>0.57</td><td>0.55</td><td>0.52</td><td>0.49</td><td>0.47</td><td>13</td><td>361</td><td>203</td></tr><tr><td>PropMatch</td><td>0.86</td><td>0.29</td><td>0.15</td><td>0.1</td><td>0.08</td><td>0</td><td>0</td><td>0</td></tr></tbody></table>

With regard to two baselines we can group tools according to system's position: six systems outperformed above both baselines (GraphMatcher, SORBETMatcher, LogMap, Matcha, ALIN, and OLaLa); three systems performed better than StringEquiv baseline (LogMapLt, AMD, and LSMatch), and two systems performed worse than both baselines (TOMATO, and PropMatch). Four matchers (AMD, ALIN, LSMatch, and SORBETMatcher) do not match properties at all. On the other side, PropMatch does not match classes at all, while it dominates in matching properties. Naturally, this has a negative effect on their overall performance.

Several systems use reference alignments to a certain extent (GraphMatcher uses it in its 5-fold cross-validation and TOMATO uses it in its sampling process) for their model learning, as has happened with some ML-based systems in the past. These systems describe their usage in their system papers.

The performance of all matching systems regarding their precision, recall and $F_1$-measure is plotted in Figure 2. Systems are represented as squares or triangles, whereas the baselines are represented as circles.

The Conference evaluation results using the uncertain reference alignments are presented in Table 11. Out of the 11 alignment systems, 6 (ALIN, AMD, LogMapLt, LSMatch, SORBETMatcher, and TOMATO) use 1.0 as the confidence value for all matches they identify. The remaining 5 systems (GraphMatcher, LogMap, Matcha, OLaLa, and PropMatch) have a wide variation of confidence values.

When comparing the performance of the systems on uncertain reference alignments versus the sharp versions, it is evident that in the discrete cases, all systems either performed at the same
<figure><img src="image_6.png" /><figcaption>**Figure 2:** Precision/recall triangular graph for the conference test case. Dotted lines depict level of precision/recall while values of F₁-measure are depicted by areas bordered by corresponding lines F₁-measure=0.[5|6|7].</figcaption></figure>

<caption>**Table 11**</caption>

F-measure, precision, and recall of the different matchers when evaluated using the sharp (ra1), discrete uncertain and continuous uncertain metrics.

<table><thead><tr><th rowspan="2">System</th><th colspan="3">Sharp</th><th colspan="3">Discrete</th><th colspan="3">Continuous</th></tr><tr><th>Prec</th><th>F-ms</th><th>Rec</th><th>Prec</th><th>F-ms</th><th>Rec</th><th>Prec</th><th>F-ms</th><th>Rec</th></tr></thead><tbody><tr><td>ALIN</td><td>0.88</td><td>0.61</td><td>0.47</td><td>0.88</td><td>0.70</td><td>0.59</td><td>0.87</td><td>0.71</td><td>0.60</td></tr><tr><td>AMD</td><td>0.87</td><td>0.58</td><td>0.43</td><td>0.87</td><td>0.66</td><td>0.53</td><td>0.86</td><td>0.67</td><td>0.55</td></tr><tr><td>GraphMatcher</td><td>0.76</td><td>0.78</td><td>0.80</td><td>0.64</td><td>0.73</td><td>0.84</td><td>0.65</td><td>0.73</td><td>0.84</td></tr><tr><td>LogMap</td><td>0.81</td><td>0.68</td><td>0.58</td><td>0.81</td><td>0.70</td><td>0.62</td><td>0.80</td><td>0.66</td><td>0.57</td></tr><tr><td>LogMapLt</td><td>0.73</td><td>0.59</td><td>0.50</td><td>0.73</td><td>0.67</td><td>0.62</td><td>0.72</td><td>0.67</td><td>0.63</td></tr><tr><td>LSMatch</td><td>0.88</td><td>0.57</td><td>0.42</td><td>0.88</td><td>0.66</td><td>0.53</td><td>0.88</td><td>0.67</td><td>0.54</td></tr><tr><td>Matcha</td><td>0.70</td><td>0.67</td><td>0.64</td><td>0.62</td><td>0.68</td><td>0.76</td><td>0.64</td><td>0.69</td><td>0.74</td></tr><tr><td>OLaLa</td><td>0.56</td><td>0.61</td><td>0.66</td><td>0.58</td><td>0.65</td><td>0.75</td><td>0.57</td><td>0.63</td><td>0.70</td></tr><tr><td>PropMatch</td><td>0.83</td><td>0.15</td><td>0.08</td><td>0.76</td><td>0.16</td><td>0.09</td><td>0.73</td><td>0.13</td><td>0.07</td></tr><tr><td>SORBETMatcher</td><td>0.78</td><td>0.70</td><td>0.64</td><td>0.71</td><td>0.71</td><td>0.72</td><td>0.69</td><td>0.70</td><td>0.72</td></tr><tr><td>TOMATO</td><td>0.61</td><td>0.55</td><td>0.61</td><td>0.61</td><td>0.62</td><td>0.63</td><td>0.60</td><td>0.62</td><td>0.64</td></tr></tbody></table>

level as before or showed improvements in terms of F-measure. The changes in F-measure for discrete cases ranged from 1 to 16 percent above those in the sharp reference alignment. Notably, LSMatch exhibited the most significant performance surge at (16%), closely followed by ALIN at (15%) and AMD at (14%). This substantial improvement was primarily driven by increased recall, a result of having fewer 'controversial' matches in the uncertain version of the reference alignment.
In contrast, systems with confidence values consistently set at 1.0 delivered very similar performance, regardless of whether a discrete or continuous evaluation methodology was applied. This was due to their proficiency in identifying matches with which experts had a high degree of agreement, while the matches they missed were typically the more contentious ones. Notably, GraphMatcher stood out by producing the highest F-measure under both continuous (73%) and discrete (73%) evaluation methodologies. This indicates that the system's confidence evaluation effectively reflects the consensus among experts in this task. However, it's worth noting that GraphMatcher experienced relatively small drops in F-measure when transitioning from discrete to continuous evaluation, primarily due to a decrease in precision.

In addition to the above findings, eight systems that participated this year were also part of the previous year's evaluation, allowing for some valuable comparisons over time. Among these, six systems demonstrated remarkable stability in their F-measures when assessed against uncertain reference alignments. However, two systems, Matcha and TOMATO, exhibited significant improvements this year. Matcha's F-measure jumped from (12%) in continuous and (14%) in discrete last year to (63%) in continuous and (65%) in discrete this year, primarily due to an increase in precision. Similarly, TOMATO saw a substantial increase in F-measure from (15%) last year to (62%) this year, both in continuous and discrete evaluations. These improvements mark significant progress for these two systems compared to the previous year.

OLaLa, PropMatch, and SORBETMatcher are three new systems participating this year. OLaLa has shown notable performance improvements, with a (4%) increase in the discrete case and a (2%) increase in the continuous case concerning F-measure when compared to the sharp reference alignment. OLaLa's F-measure has risen from (61%) to (65%) in the discrete case and to (63%) in the continuous case. This improvement is primarily attributed to an increase in recall.

On the other hand, PropMatch and SORBETMatcher have demonstrated similar performance in both discrete and continuous cases when compared to the sharp reference alignment in terms of F-measure. Notably, PropMatch exhibits consistently lower precision and recall across the three different versions of the reference alignment, primarily due to its narrow focus on property matching.

## 4.4. Multifarm

This year, 4 systems have registered to participate in the Multifarm track:LSMatch-Multilingual, LogMap, LogMapLt, and Matcha. The number of participating tools is stable with respect to the last 4 campaigns (5 in 2022, 6 in 2021, 6 in 2020, 5 in 2019, 6 in 2018, 8 in 2017, 7 in 2016, 5 in 2015, 3 in 2014, 7 in 2013, and 7 in 2012). This year, we lost the participation of CIDER-LM. The reader can refer to the OAEI papers for a detailed description of the strategies adopted by each system.

The Multifarm evaluation results based on the blind dataset are presented in Table 12, demonstrating the aggregated results for the matching tasks. They have been computed using the MELT framework without applying any threshold to the results. They are measured in terms of macro precision and recall. The results of non-specific systems are not reported here, as we could observe in the last campaigns that they can have intermediate results in tests of type ii) (same ontologies task) and poor performance in tests i) (different ontologies task).

The systems have been executed on a Ubuntu Linux machine configured with 32GB of RAM
**Table 12**

<table><caption>Multifarm aggregated results per matcher, for each type of matching task – different ontologies. Time is measured in minutes.</caption><thead><tr><th rowspan="2">System</th><th colspan="4">Different ontologies (i)</th></tr><tr><th>Time(Min)</th><th>Prec.</th><th>F-m.</th><th>Rec.</th></tr></thead><tbody><tr><td>Matcha</td><td>2</td><td>.37</td><td>.08</td><td>.04</td></tr><tr><td>LSMatch-Multilingual</td><td>46</td><td>.68</td><td>.47</td><td>.36</td></tr><tr><td>LogMap</td><td>9</td><td>.72</td><td>.44</td><td>.31</td></tr><tr><td>LogMapLt</td><td>227</td><td>.24</td><td>.038</td><td>.02</td></tr></tbody></table>

under an Intel Core CPU 2.00GHz x8 processors. All measurements are based on a single run. As for each campaign, we observed large differences in the time required for a system to complete the 55 x 24 matching tasks: LSMatch-Multilingual (46 min), LogMap (9 minutes), LogMapLt (227 minutes), and Matcha (2 minutes). When we compare the times to last year's campaign, we can see that LogMap has a stable 9 min execution, whereas LogMapLt saw a decline in timing from 175 min to 227 min, and LSMatch-Multilingual improved the timing from 69 min to 46 min. Since the other tools are participating for the first time, their timings are incomparable. These measurements only indicate the time the systems require to finish the task in a common environment. LSMatch-Multilingual outperforms all other systems in terms of F-measure (0.47) and recall (0.36), and LogMap outperforms all other systems in terms of Precision (0.72).

It is seen that a similar number of systems have participated in the campaign through the years. However, there is a dynamicity of the tools, such that, each year, participating tools vary. In 2023, we had four systems participating in the campaign where 1 of them was a new system, and 3 were long-term participating systems. As observed in several campaigns, still, all systems still privilege precision in detriment to recall (recall below 0.50), and the results are below the ones obtained for the Conference's original dataset.

## 4.5. Complex Matching

Unfortunately, this track has not attracted many participants in the last two years. This year, MatchaC, LogMap and LogMapLt have been registered to participate. While LogMapLt and LogMap are dedicated to generating simple correspondences, only LogMap was able to generate non-empty alignments. MatchaC, the only system specifically designed to generate expressive correspondences, had some problems dealing with the datasets and was not able to generate any valid alignment.

## 4.6. Food

This is the second year of the track and five systems were registered: AMD, LogMap, LogMapLt, OLaLa and Matcha. AMD encountered an error during launch, so evaluation results only available for four of the matchers are presented in Table 13.

The test case food v2 evaluates matching systems regarding their capability to find “equal” (=) and “subclass” relation (<) correspondences between the CIQUAL ontology and the SIREN ontology. All evaluated systems compute the alignment in less than a minute except OLaLa.
<table><caption>**Table 13**
Food track results per matcher. Time is measured in seconds.</caption><thead><tr><th>System</th><th>Corresp.</th><th>Precision</th><th>Recall</th><th>F1-measure</th><th>Time(s)</th></tr></thead><tbody><tr><td colspan="6">"equal" (=) relation</td></tr><tr><td>OLaLa</td><td>1185</td><td>0.0337</td><td>0.5479</td><td>0.06359</td><td>73920</td></tr><tr><td>LogMap</td><td>15</td><td>0.1333</td><td>0.0274</td><td>0.0454</td><td>14</td></tr><tr><td>LogMapLt</td><td>15</td><td>0.1333</td><td>0.0274</td><td>0.0454</td><td>5</td></tr><tr><td>Matcha</td><td>335</td><td>0.0656</td><td>0.3013</td><td>0.1078</td><td>53</td></tr><tr><td colspan="6">"subclass" relation (&lt;) relation</td></tr><tr><td>LogMap</td><td>15</td><td>0</td><td>0</td><td>0</td><td>15</td></tr><tr><td>LogMapLt</td><td>15</td><td>0</td><td>0</td><td>0</td><td>6</td></tr><tr><td>Matcha</td><td>335</td><td>0</td><td>0</td><td>0</td><td>53</td></tr></tbody></table>

LogMapLt stands out for its very fast calculation time of 5s (resp. 6s) to find "equal" (resp. "subclass" relation correspondences). Concerning "equal" (=) relation correspondences, LogMap and LogMapLt have better precision than Matcha and OLaLa. However, LogMap's recall is 20 (resp. 11 times) less than OLaLa's (resp. Matcha's) one. Matcha is the best-performing participant in the FNC test case in terms of precision and F1-measure. None of the matching systems are able to find "subclass" relation (<) correspondences.

## 4.7. Interactive matching

This year, two systems (ALIN, and LogMap) participated in the Interactive matching track. Their results are shown in Table 14 and Figure 3 for both the Anatomy and Conference datasets.

The table includes the following information (column names within parentheses):

- The performance of the system: Precision (Prec.), Recall (Rec.), and F-measure (F-m.) with respect to the fixed reference alignment, as well as Recall+ (Rec.+) for the Anatomy task. To facilitate the assessment of the impact of user interactions, we also provide the performance results from the original tracks, without interaction (line with Error NI).

- To ascertain the impact of the oracle errors, we provide the performance of the system with respect to the oracle (i.e., the reference alignment as modified by the errors introduced by the oracle: Precision oracle (Prec. oracle), Recall oracle (Rec. oracle) and F-measure oracle (F-m. oracle). For a perfect oracle, these values match the actual performance of the system.

- Total requests (Tot Reqs.) represents the number of distinct user interactions with the tool, where each interaction can contain one to three conflicting correspondences, that could be analyzed simultaneously by a user.

- Distinct correspondences (Dist. Mapps) counts the total number of correspondences for which the oracle gave feedback to the user (regardless of whether they were submitted simultaneously, or separately).
<table><caption>**Table 14**
Interactive matching results for the Anatomy and Conference datasets.</caption><thead><tr><th>Tool</th><th>Error</th><th>Prec.</th><th>Rec.</th><th>F-m.</th><th>Rec.+</th><th>Prec. oracle</th><th>Rec. oracle</th><th>F-m. oracle</th><th>Tot. Reqs.</th><th>Dist. Mapps</th><th>Pos. Prec.</th><th>Neg. Prec.</th></tr></thead><tbody><tr><td colspan="14" style="text-align:center; font-weight:bold;">Anatomy Dataset</td></tr><tr><td rowspan="4">ALIN</td><td>NI</td><td>0.983</td><td>0.726</td><td>0.835</td><td>0.438</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>0.0</td><td>0.987</td><td>0.92</td><td>0.952</td><td>0.787</td><td>0.987</td><td>0.92</td><td>0.952</td><td>514</td><td>1453</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>0.1</td><td>0.91</td><td>0.9</td><td>0.904</td><td>0.759</td><td>0.987</td><td>0.922</td><td>0.953</td><td>491</td><td>1390</td><td>0.666</td><td>0.975</td><td></td></tr><tr><td>0.2</td><td>0.846</td><td>0.883</td><td>0.864</td><td>0.73</td><td>0.988</td><td>0.924</td><td>0.955</td><td>476</td><td>1352</td><td>0.469</td><td>0.947</td><td></td></tr><tr><td>0.3</td><td>0.792</td><td>0.864</td><td>0.826</td><td>0.699</td><td>0.988</td><td>0.925</td><td>0.955</td><td>466</td><td>1329</td><td>0.342</td><td>0.912</td><td></td></tr><tr><td rowspan="4">LogMap</td><td>NI</td><td>0.915</td><td>0.848</td><td>0.88</td><td>0.602</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>0.0</td><td>0.988</td><td>0.846</td><td>0.912</td><td>0.595</td><td>0.988</td><td>0.846</td><td>0.912</td><td>388</td><td>1164</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>0.1</td><td>0.966</td><td>0.831</td><td>0.894</td><td>0.569</td><td>0.971</td><td>0.803</td><td>0.879</td><td>388</td><td>1164</td><td>0.75</td><td>0.967</td><td></td></tr><tr><td>0.2</td><td>0.952</td><td>0.822</td><td>0.882</td><td>0.549</td><td>0.951</td><td>0.764</td><td>0.847</td><td>388</td><td>1164</td><td>0.573</td><td>0.925</td><td></td></tr><tr><td>0.3</td><td>0.936</td><td>0.819</td><td>0.873</td><td>0.545</td><td>0.925</td><td>0.723</td><td>0.812</td><td>388</td><td>1164</td><td>0.43</td><td>0.876</td><td></td></tr><tr><td colspan="14" style="text-align:center; font-weight:bold;">Conference Dataset</td></tr><tr><td rowspan="4">ALIN</td><td>NI</td><td>0.874</td><td>0.456</td><td>0.599</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>0.0</td><td>0.919</td><td>0.744</td><td>0.822</td><td>-</td><td>0.919</td><td>0.744</td><td>0.822</td><td>274</td><td>815</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>0.1</td><td>0.705</td><td>0.7</td><td>0.702</td><td>-</td><td>0.934</td><td>0.773</td><td>0.846</td><td>264</td><td>785</td><td>0.504</td><td>0.988</td><td></td></tr><tr><td>0.2</td><td>0.569</td><td>0.663</td><td>0.612</td><td>-</td><td>0.944</td><td>0.796</td><td>0.863</td><td>291</td><td>764</td><td>0.305</td><td>0.977</td><td></td></tr><tr><td>0.3</td><td>0.462</td><td>0.634</td><td>0.535</td><td>-</td><td>0.952</td><td>0.816</td><td>0.879</td><td>250</td><td>743</td><td>0.196</td><td>0.955</td><td></td></tr><tr><td rowspan="4">LogMap</td><td>NI</td><td>0.801</td><td>0.58</td><td>0.67</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>0.0</td><td>0.886</td><td>0.61</td><td>0.723</td><td>-</td><td>0.886</td><td>0.61</td><td>0.723</td><td>82</td><td>246</td><td>1.0</td><td>1.0</td><td></td></tr><tr><td>0.1</td><td>0.851</td><td>0.599</td><td>0.703</td><td>-</td><td>0.862</td><td>0.577</td><td>0.691</td><td>82</td><td>246</td><td>0.703</td><td>0.983</td><td></td></tr><tr><td>0.2</td><td>0.824</td><td>0.592</td><td>0.689</td><td>-</td><td>0.841</td><td>0.551</td><td>0.666</td><td>82</td><td>246</td><td>0.526</td><td>0.952</td><td></td></tr><tr><td>0.3</td><td>0.793</td><td>0.581</td><td>0.671</td><td>-</td><td>0.804</td><td>0.519</td><td>0.631</td><td>82</td><td>246</td><td>0.36</td><td>0.897</td><td></td></tr></tbody></table>

NI stands for non-interactive, and refers to the results obtained by the matching system in the original track.

- Finally, the performance of the oracle itself with respect to the errors it introduced can be gauged through the positive precision (Pos. Prec.) and negative precision (Neg. Prec.), which measure respectively the fraction of positive and negative answers given by the oracle that are correct. For a perfect oracle, these values are equal to 1 (or 0, if no questions were asked).

The figure shows the time intervals between the questions to the user/oracle for the different systems and error rates. Different runs are depicted with different colors.

The matching systems that participated in this track employ different user-interaction strategies. While LogMap makes use of user interactions exclusively in the post-matching steps to filter their candidate correspondences, ALIN can also add new candidate correspondences to its initial set. LogMap requests feedback on only selected correspondences candidates (based on their similarity patterns or their involvement in unsatisfiabilities). ALIN and LogMap can both ask the oracle to analyze several conflicting correspondences simultaneously.

The performance of the systems usually improves when interacting with a perfect oracle in comparison with no interaction. ALIN is the system that improves the most, because of its high
<figure><img src="image_7.png" /><figcaption>**Figure 3:** Time intervals between requests to the user/oracle for the Anatomy (top 4 plots) and Conference (bottom 4 plots) datasets. Whiskers: Q1-1,5IQR, Q3+1,5IQR, IQR=Q3-Q1. The labels under the system names show the average number of requests and the mean time between the requests for the ten runs.</figcaption></figure>
number of oracle requests, and its non-interactive performance was the lowest of the interactive
systems, and thus the easiest to improve.

Although system performance deteriorates when the error rate increases, there are still benefits
from the user interaction—some of the systems’ measures stay above their non-interactive values
even for the larger error rates. Naturally, the more a system relies on the oracle, the more its
performance tends to be affected by the oracle’s errors.

The impact of the oracle's errors is linear for ALIN in most tasks, as the F-measure according to the oracle remains approximately constant across all error rates. It is supra-linear for LogMap in all datasets.

Another aspect that was assessed, was the response time of systems, i.e., the time between requests. Two models for system response times are frequently used in the literature [60]: Shneiderman and Seow take different approaches to categorize the response times taking a task-centered view and a user-centered view respectively. According to task complexity, Shneiderman defines response time in four categories: typing, mouse movement (50-150 ms), simple frequent tasks (1 s), common tasks (2-4 s) and complex tasks (8-12 s). While Seow's definition of response time is based on the user expectations towards the execution of a task: instantaneous (100-200 ms), immediate (0.5-1 s), continuous (2-5 s), captive (7-10 s). Ontology alignment is a cognitively demanding task and can fall into the third or fourth categories in both models. In this regard the response times (request intervals as we call them above) observed in all datasets fall into the tolerable and acceptable response times, and even into the first categories, in both models. The request intervals for LogMap and ALIN stay at a few milliseconds for most datasets. It could be the case, however, that a user would not be able to take advantage of these low response times because the task complexity may result in higher user response time (i.e., the time the user needs to respond to the system after the system is ready).

4.8. Bio-ML

Our results include five tables for equivalence matching, five tables for subsumption matching, and two tables for Bio-LLM, where each table corresponds to an OM pair and includes results of both the unsupervised and semi-supervised settings. For the full results, please refer to the OAEI 2023 Bio-ML website<fnref n="32" />.

Briefly, we have the following participants for equivalence matching: (i) machine learning-based systems including BERTMap, BERTMapLt [58], AMD [61], Matcha, Matcha-DL [62], OLala, and SORBETMatcher [63]; and (ii) traditional systems including LogMap, LogMapBio, and LogMapLt [36]. For equivalence matching, top performing systems vary across different tasks, with LogMapBio attaining the best F1 on 2 out of 5 unsupervised tasks, and AMD, BERTMap, and SORBETMatcher attaining the best F1 on each of the remaining three, respectively. BERTMap also attains the best ranking scores of all tasks, though most systems do not provide ranking results in equivalence matching. For subsumption matching, all the participating systems are machine learning-based, including BERTSubs (IC) [59], OWL2Vec*+RF [64], SORBETMatcher [63], and Word2Vec+Random Forest (RF). SORBETMatcher attains the best MRR on 3 out of 5 subsumption tasks, while BERTSubs (IC) and OWL2Vec*+RF attain the best MRR on each of

<footnote n="32">³²https://www.cs.ox.ac.uk/isg/projects/ConCur/oaei/2023/, permanent link at Internet Wayback Machine, https://web.archive.org/web/20231120093331/https://www.cs.ox.ac.uk/isg/projects/ConCur/oaei/2023/</footnote>
the remaining two, respectively. Overall, the 2023 edition attracted more machine learning-based participants, which matches the original purpose of Bio-ML, while LogMap variants are the only symbolic participants.

## 4.9. Biodiversity and Ecology

This year, five matching systems (LogMap, LogMapLt, LogMapKG, Matcha and OLaLa) managed to generate an output for all of the track tasks, except Matcha failed to achieve alignment for the envo-sweet task. As in previous editions, we used precision, recall, and F-measure to evaluate the performance of the participating systems. The results for the Biodiversity and Ecology track are shown in Table 15.

In comparison to the previous year, a smaller number of systems succeeded in generating alignments for the track tasks. The results of the participating systems are comparable to last year in terms of F-measure. In terms of run time, OLaLa took the longer. Regarding the ENVO-SWEET task, only OLaLa and the LogMap family systems achieved it with a similar performance to last year. The MACROALGAE-MACROZOOBENTHOS and FISH-ZOOPLANKTON matching tasks involve resources developed in SKOS. For the transformation, we made use of a source code directly derived from the AML ontology parsing module, kindly provided to us by its developers. The systems that did not perform well in this task did map a large number of dissimilar concepts that happen to have similar URIs. All systems performed well on most NCBITAXON-TAXREF-LD subtasks, with slightly the same levels of precision and recall. Overall, in this year's evaluation, the number of participating systems decreased and the performance of the successful ones remained similar.

## 4.10. Material Sciences and Engineering (MSE)

This year four systems registered on the MSE track, each of which was used for evaluation with the three test cases of the MSE benchmark. AMD produced errors and an empty alignment file, so results are only available for three of the matchers: LogMap, LogMapLt, Matcha. The evaluation results are shown in Table 16.

The first test case evaluates matching systems regarding their capability to find “equal” (=), “superclass” (>) and “subclass” (<) correspondences between the mid-sized MatOnto and the small-sized (since reduced) MaterialInformation ontology. None of the evaluated systems finds correspondences other than “equal” (=). All evaluated systems compute the alignment in less than a minute. In contrast to the results of 2022 Matcha performs the matching task almost as quickly as LogMap, which stood out for its very fast calculation time. Apart from the runtime, the results for LogMap and LogMapLt do not change in comparison to the evaluation in 2022. LogMap presents a maximum precision value of 1.0, however since only one correspondence was found by LogMap, the recall and hence the F1-measure is low (0.083). In direct comparison to LogMap, LogMapLt calculates the alignment in around half the time and achieves much lower precision (0.4) but due to a greater amount of correctly found correspondences, the F1-measure is better - although still low with 0.143. Matcha finds 8 incorrect correspondences and thus is the worst-performing participant in terms of precision in the first test case. Investigating the reason for this low precision, Matcha appears to match classes with object
**Table 15**

Results for the Biodiversity & Ecology track.

<table><thead><tr><th>System</th><th>Time (HH:MM:SS)</th><th>Number of mappings</th><th>Precision</th><th>Recall</th><th>F-measure</th></tr></thead><tbody><tr><td colspan="6">ENVO-SWEET task</td></tr><tr><td>LogMap</td><td>00:00:36</td><td>681</td><td>0.780</td><td>0.655</td><td>0.713</td></tr><tr><td>LogMapKG</td><td>00:00:28</td><td>677</td><td>0.781</td><td>0.657</td><td>0.714</td></tr><tr><td>LogMapLt</td><td>00:05:40</td><td>595</td><td>0.829</td><td>0.594</td><td>0.692</td></tr><tr><td>OLaLa</td><td>06:46:18</td><td>1081</td><td>0.484</td><td>0.650</td><td>0.555</td></tr><tr><td colspan="6">MACROALGAE-MACROZOOBENTHOS task</td></tr><tr><td>LogMap</td><td>00:00:03</td><td>29</td><td>0.275</td><td>0.444</td><td>0.340</td></tr><tr><td>LogMapKG</td><td>00:00:04</td><td>29</td><td>0.275</td><td>0.444</td><td>0.340</td></tr><tr><td>LogMapLt</td><td>00:00:00</td><td>9</td><td>0.857</td><td>0.333</td><td>0.480</td></tr><tr><td>OLaLa</td><td>00:08:30</td><td>10</td><td>0.7</td><td>0.388</td><td>0.5</td></tr><tr><td>Matcha</td><td>00:00:07</td><td>45</td><td>0.2</td><td>0.5</td><td>0.285</td></tr><tr><td colspan="6">FISH-ZOOPLANKTON task</td></tr><tr><td>LogMap</td><td>00:00:03</td><td>32</td><td>0.093</td><td>0.2</td><td>0.127</td></tr><tr><td>LogMapKG</td><td>00:00:04</td><td>55</td><td>0.218</td><td>0.8</td><td>0.342</td></tr><tr><td>LogMapLt</td><td>00:00:00</td><td>8</td><td>1.0</td><td>0.533</td><td>0.695</td></tr><tr><td>OLaLa</td><td>00:07:59</td><td>13</td><td>1.0</td><td>0.866</td><td>0.928</td></tr><tr><td>Matcha</td><td>00:00:11</td><td>47</td><td>0.276</td><td>0.866</td><td>0.419</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Animalia task</td></tr><tr><td>LogMap</td><td>00:00:43</td><td>72899</td><td>0.660</td><td>0.998</td><td>0.795</td></tr><tr><td>LogMapKG</td><td>00:11:32</td><td>72898</td><td>0.660</td><td>0.998</td><td>0.795</td></tr><tr><td>LogMapLt</td><td>00:00:43</td><td>72010</td><td>0.665</td><td>0.993</td><td>0.796</td></tr><tr><td>OLaLa</td><td>68:27:32</td><td>70821</td><td>0.679</td><td>0.998</td><td>0.808</td></tr><tr><td>Matcha</td><td>00:04:18</td><td>71008</td><td>0.674</td><td>0.993</td><td>0.803</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Bacteria task</td></tr><tr><td>LogMap</td><td>00:00:01</td><td>304</td><td>0.575</td><td>1.0</td><td>0.730</td></tr><tr><td>LogMapKG</td><td>00:00:01</td><td>304</td><td>0.575</td><td>1.0</td><td>0.730</td></tr><tr><td>LogMapLt</td><td>00:00:00</td><td>290</td><td>0.6</td><td>0.994</td><td>0.748</td></tr><tr><td>OLaLa</td><td>00:19:32</td><td>294</td><td>0.593</td><td>0.994</td><td>0.743</td></tr><tr><td>Matcha</td><td>00:00:14</td><td>300</td><td>0.58</td><td>0.994</td><td>0.732</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Chromista task</td></tr><tr><td>LogMap</td><td>00:00:04</td><td>2218</td><td>0.623</td><td>0.985</td><td>0.764</td></tr><tr><td>LogMapKG</td><td>00:00:01</td><td>2218</td><td>0.623</td><td>0.985</td><td>0.764</td></tr><tr><td>LogMapLt</td><td>00:00:01</td><td>2165</td><td>0.637</td><td>0.982</td><td>0.773</td></tr><tr><td>OLaLa</td><td>01:59:05</td><td>2173</td><td>0.634</td><td>0.981</td><td>0.771</td></tr><tr><td>Matcha</td><td>00:00:48</td><td>2213</td><td>0.624</td><td>0.984</td><td>0.764</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Fungi task</td></tr><tr><td>LogMap</td><td>00:00:39</td><td>12949</td><td>0.783</td><td>0.998</td><td>0.878</td></tr><tr><td>LogMapKG</td><td>00:00:40</td><td>12949</td><td>0.783</td><td>0.998</td><td>0.878</td></tr><tr><td>LogMapLt</td><td>00:00:07</td><td>12929</td><td>0.783</td><td>0.997</td><td>0.877</td></tr><tr><td>OLaLa</td><td>11:54:37</td><td>12549</td><td>0.807</td><td>0.996</td><td>0.891</td></tr><tr><td>Matcha</td><td>00:01:43</td><td>12925</td><td>0.785</td><td>0.998</td><td>0.879</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Plantae task</td></tr><tr><td>LogMap</td><td>00:01:44</td><td>26912</td><td>0.731</td><td>0.988</td><td>0.840</td></tr><tr><td>LogMapKG</td><td>00:01:36</td><td>26910</td><td>0.731</td><td>0.988</td><td>0.840</td></tr><tr><td>LogMapLt</td><td>00:00:17</td><td>26359</td><td>0.746</td><td>0.987</td><td>0.849</td></tr><tr><td>OLaLa</td><td>22:04:48</td><td>25667</td><td>0.769</td><td>0.991</td><td>0.866</td></tr><tr><td>Matcha</td><td>00:03:16</td><td>26597</td><td>0.741</td><td>0.989</td><td>0.847</td></tr><tr><td colspan="6">NCBITAXON-TAXREFLD Protozoa task</td></tr><tr><td>LogMap</td><td>00:00:01</td><td>496</td><td>0.719</td><td>1.0</td><td>0.837</td></tr><tr><td>LogMapKG</td><td>00:00:01</td><td>496</td><td>0.719</td><td>1.0</td><td>0.837</td></tr><tr><td>LogMapLt</td><td>00:00:00</td><td>477</td><td>0.746</td><td>0.997</td><td>0.853</td></tr><tr><td>OLaLa</td><td>00:32:17</td><td>476</td><td>0.750</td><td>1.0</td><td>0.857</td></tr><tr><td>Matcha</td><td>00:00:44</td><td>493</td><td>0.724</td><td>1.0</td><td>0.840</td></tr></tbody></table>
<table><caption>**Table 16**
Results for the three test cases of the MSE track.</caption><thead><tr><th>System</th><th>Corresp.</th><th>Precision</th><th>Recall</th><th>F1-Measure</th><th>Time [s]</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">First Test Case</td></tr><tr><td>LogMap</td><td>1</td><td>1.000</td><td>0.043</td><td>0.083</td><td>20</td></tr><tr><td>LogMapLt</td><td>5</td><td>0.400</td><td>0.087</td><td>0.143</td><td>53</td></tr><tr><td>Matcha</td><td>11</td><td>0.273</td><td>0.130</td><td>0.176</td><td>28</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">Second Test Case</td></tr><tr><td>LogMap</td><td>67</td><td>0.881</td><td>0.195</td><td>0.320</td><td>6</td></tr><tr><td>LogMapLt</td><td>67</td><td>0.851</td><td>0.189</td><td>0.309</td><td>77</td></tr><tr><td>Matcha</td><td>87</td><td>0.756</td><td>0.219</td><td>0.339</td><td>15</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">Third Test Case</td></tr><tr><td>LogMap</td><td>56</td><td>0.946</td><td>0.841</td><td>0.891</td><td>25</td></tr><tr><td>LogMapLt</td><td>56</td><td>0.911</td><td>0.810</td><td>0.857</td><td>53</td></tr><tr><td>Matcha</td><td>59</td><td>0.949</td><td>0.889</td><td>0.918</td><td>23</td></tr></tbody></table>

properties, e.g. "Temperature" = "hasTemperature" as in 2022. Since the recall is the best of the participating systems, Matcha turns out to be the best-performing system based on its F1-measure.

The second test case evaluates the matching systems to find correspondences between the large-sized MaterialInformation and the mid-sized BFO-based MatOnto. Surprisingly, LogMap performs the matching task significantly quicker than in the first test case and stands out for its very fast computation time of only 6s at a high precision of 0.881. Apart from the runtime, the results for LogMap and LogMapLt do not change in comparison to the evaluation in 2022. Since LogMap found only 59 correct correspondences out of the 302 reference correspondences, the recall is rather low, but the F1-measure is still the highest of the tested systems. LogMapLt is significantly slower than LogMap but finds the same amount of correspondences with 2 additional false positives, so it achieves a slightly lower overall F1-measure than LogMap. Matcha finds 6 wrong correspondences where classes are matched to object properties as in the first test case. Matcha presents the lowest precision in this test case but the highest recall. Based on its F1-measure, Matcha performs slightly better than the other systems in this test case.

The third test case evaluates matching systems to find correspondences between the large-sized MaterialInformation and the mid-sized EMMO. All evaluated systems compute the alignments in under one minute. Apart from the runtime, the results for LogMap and LogMapLt do not change in comparison to the evaluation in 2022. All of the systems present high precision values. LogMap computes 3 false positives, LogMapLt computes 5 false positives and Matcha 3 false positives. At the same time Matcha computes with 56 the highest number of true positives, which results in the best precision of the tested systems. Since it has the highest number of correctly found correspondences, the recall and the F1-measure are the highest of the evaluated systems in the third test case at the fastest computation time.

In summary, LogMap stands out for its very fast computing speed with very high precision at
the same time. LogMapLt is significantly slower in every test case and almost constantly shows worse results - only in the first test case the recall of LogMapLt is higher than for LogMap. In our opinion, LogMap is definitely recommended for MSE applications where high precision is demanded. In comparison to that, LogMapLt does not appear to bring any decisive advantage over LogMap.

Matcha in its current implementation is not recommended for MSE applications since it
matches classes to properties.

A-LIOn produces moderate results but does not bring any advantage over LogMap. Further-
more, A-LIOn produces errors while reasoning on EMMO. The latter is the only one of the MSE
ontologies used with a significant proportion of essential axioms. According to the annotations in
EMMO, this ontology exclusively can be inferred with the FaCT++ reasoner. That might be a
cause for the occurring reasoning errors of A-LIOn and bad results in the third test case.

None of the evaluated matcher finds all reference correspondences correctly and none of the
matchers.

4.11. Common Knowledge Graphs

We evaluated all the participating systems that were packaged as SEALS packages or as web
services using Docker (even those not registered to participate on this new track). However, not
all systems were able to complete the task, as some systems finished with an empty alignment
file. Here, we include the results of 7 systems that were able to finish the task within the 24-hour
time limit with a non-empty alignment file: LogMap, OLaLa, Matcha, LogMapLt, LogMapKG,
LsMatch, and AMD.

Table 17 shows the aggregated results on the two datasets for systems that produced non-empty
alignment files. The size column indicates the total number of class alignments discovered
by each system. While the majority of the systems discovered alignments at both schema and
instance levels, we have only evaluated class alignments, as the two gold standard does not
include any instance-level ground truth. Further, Not all systems were able to handle the original
dataset versions (i.e., those with all annotated instances). In terms of the NELL-DBpedia test
case, LogMap, OLaLa, Matcha, and AMD were able to generate results when applied to the
full-size dataset. While on the YAGO-Wikidata dataset, which is large-scale compared to the
first dataset, only OLaLa, Matcha, and AMD were able to generate alignments with the original
dataset. Other systems either fail to complete the task within the allocated 24-hour time limit
such as LogMapLt and LsMatch, or produce an empty alignment file such as LogMap (only on
the YagoWikidata dataset). LogMapKG on the other hand tends to only align instances when it is
applied to full-size datasets. Similar to the 2022 evaluation results, AMD does generate schema
alignments but in the wrong format, therefore, they can not be evaluated.

The resulted alignment files from all the participating systems are available to download on the track's result webpage³³. On the Nell-DBpedia dataset, all systems were able to outperform the basic string matcher, in terms of f-measure, except for LogMapLt. On the YagoWikidata dataset, two systems were not able to outperform the baseline, which are LogMapLt and LsMatch. This year saw the return of different matchers and the introduction of a new one, OLaLa. While most

³³https://oaei.ontologymatching.org/2023/results/commonKG/index.html
<table><caption>**Table 17**
Results for the Common Knowledge Graphs track.</caption><thead><tr><th>Matcher</th><th>Size</th><th>Precision</th><th>Recall</th><th>F1 measure</th><th>Time</th><th>Dataset Size</th></tr></thead><tbody><tr><td colspan="7" style="text-align: center; font-weight: bold;">Nell-DBpedia</td></tr><tr><td>LogMap</td><td>105</td><td>0.99</td><td>0.80</td><td>0.88</td><td>00:03:17</td><td>original</td></tr><tr><td>OLaLa</td><td>120</td><td>1.0</td><td>0.92</td><td>0.96</td><td>00:07:07</td><td>original</td></tr><tr><td>LogMapLt</td><td>77</td><td>1.00</td><td>0.60</td><td>0.75</td><td>00:26:19</td><td>small</td></tr><tr><td>LogMapKG</td><td>104</td><td>0.98</td><td>0.80</td><td>0.88</td><td>00:00:00</td><td>small</td></tr><tr><td>AMD</td><td>102</td><td>0.00</td><td>0.00</td><td>0.00</td><td>00:00:23</td><td>original</td></tr><tr><td>LsMatch</td><td>101</td><td>0.96</td><td>0.75</td><td>0.84</td><td>00:00:52</td><td>small</td></tr><tr><td>Matcha</td><td>114</td><td>0.99</td><td>0.87</td><td>0.93</td><td>00:01:53</td><td>original</td></tr><tr><td>String Baseline</td><td>78</td><td>1.00</td><td>0.60</td><td>0.75</td><td>00:00:37</td><td>original</td></tr><tr><td colspan="7" style="text-align: center; font-weight: bold;">Yago-Wikidata</td></tr><tr><td>LogMap</td><td>233</td><td>1.00</td><td>0.76</td><td>0.86</td><td>00:00:26</td><td>small</td></tr><tr><td>OLaLa</td><td>209</td><td>1.0</td><td>0.68</td><td>0.81</td><td>00:03:56</td><td>original</td></tr><tr><td>LogMapLt</td><td>211</td><td>1.00</td><td>0.70</td><td>0.81</td><td>00:54:13</td><td>small</td></tr><tr><td>LogMapKG</td><td>232</td><td>1.00</td><td>0.76</td><td>0.83</td><td>00:00:10</td><td>small</td></tr><tr><td>AMD</td><td>125</td><td>0.00</td><td>0.00</td><td>0.00</td><td>00:29:04</td><td>original</td></tr><tr><td>LsMatch</td><td>196</td><td>0.97</td><td>0.63</td><td>0.76</td><td>00:02:33</td><td>small</td></tr><tr><td>Matcha</td><td>274</td><td>0.99</td><td>0.90</td><td>0.94</td><td>00:07:16</td><td>original</td></tr><tr><td>String Baseline</td><td>212</td><td>1.00</td><td>0.70</td><td>0.82</td><td>00:00:02</td><td>original</td></tr></tbody></table>

matchers demonstrated similar performance to previous evaluations, Matcha notably improved its results on both datasets. Matcha also showcased the ability to function with the original datasets, a capability it lacked in the 2022 evaluation. OLaLa outperformed all other matchers in the Nell-DBpedia task, whereas Matcha excelled on the larger dataset, Yago-Wikidata. Furthermore, all matching processes were completed in less than an hour, as indicated in the runtime column. Lastly, the dataset size column specifies whether a system operated on the original dataset or solely on the smaller version.

## 4.12. Crosswalks Data Schema Matching

All the systems registered to OAEI 2023 were run besides the fact that only LogMap has been registered to participate in all tracks and no system has been specifically registered to the Crosswalks task.

This year, as introduced above, we have used the schemes for which an OWL/RDFS serialization is available, as OAEI matching systems are used to the format. However, this does not reflect the reality of the field, as schemes are not usually exposed in such a structured format. This opens the possibility of providing a dedicated task next year.

Table 18 shows the results for the systems that have generated correspondences. While generating a few number of correct correspondences, precision is higher with respect to recall for
<table><caption>**Table 18**
Results for the Crosswalks task.</caption><thead><tr><th></th><th>Correct</th><th>Output</th><th>Expected</th><th>Precision</th><th>Recall</th></tr></thead><tbody><tr><td colspan="6" style="text-align: center; font-weight: bold;">LogMap</td></tr><tr><td>datacity</td><td>0</td><td>3</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td>iso</td><td>0</td><td>4</td><td>42</td><td>0.00</td><td>0.00</td></tr><tr><td>dcat3</td><td>1</td><td>12</td><td>42</td><td>0.08</td><td>0.02</td></tr><tr><td>dcterms</td><td>0</td><td>1</td><td>32</td><td>0.00</td><td>0.00</td></tr><tr><td>dcat-ap</td><td>0</td><td>2</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td></td><td>1</td><td>22</td><td>184</td><td>0.02</td><td>0.00</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">LogMapBio</td></tr><tr><td>datacity</td><td>0</td><td>0</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td>iso</td><td>0</td><td>0</td><td>42</td><td>0.00</td><td>0.00</td></tr><tr><td>dcat3</td><td>1</td><td>12</td><td>42</td><td>0.08</td><td>0.02</td></tr><tr><td>dcterms</td><td>1</td><td>1</td><td>32</td><td>1.00</td><td>0.03</td></tr><tr><td>dcat-ap</td><td>0</td><td>2</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td></td><td>2</td><td>15</td><td>184</td><td>0.22</td><td>0.01</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">LogMapKG</td></tr><tr><td>datacity</td><td>0</td><td>0</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td>iso</td><td>0</td><td>4</td><td>42</td><td>0.00</td><td>0.00</td></tr><tr><td>dcat</td><td>31</td><td>12</td><td>42</td><td>0.08</td><td>0.02</td></tr><tr><td>dcterms</td><td>0</td><td>1</td><td>32</td><td>0.00</td><td>0.00</td></tr><tr><td>dcat-ap</td><td>0</td><td>2</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td></td><td>1</td><td>19</td><td>184</td><td>0.02</td><td>0.00</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">LogMapLt</td></tr><tr><td>datacity</td><td>0</td><td>4</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td>iso</td><td>1</td><td>14</td><td>42</td><td>0.07</td><td>0.02</td></tr><tr><td>dcat3</td><td>4</td><td>41</td><td>42</td><td>0.10</td><td>0.10</td></tr><tr><td>dcterms</td><td>1</td><td>3</td><td>32</td><td>0.33</td><td>0.03</td></tr><tr><td>dcat-ap</td><td>0</td><td>4</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td></td><td>6</td><td>66</td><td>184</td><td>0.10</td><td>0.03</td></tr><tr><td colspan="6" style="text-align: center; font-weight: bold;">Matcha</td></tr><tr><td>datacity</td><td>0</td><td>5</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td>iso</td><td>1</td><td>10</td><td>42</td><td>0.10</td><td>0.02</td></tr><tr><td>dcat3</td><td>4</td><td>44</td><td>42</td><td>0.09</td><td>0.10</td></tr><tr><td>dcterms</td><td>1</td><td>1</td><td>32</td><td>1.00</td><td>0.03</td></tr><tr><td>dcat-ap</td><td>0</td><td>0</td><td>34</td><td>0.00</td><td>0.00</td></tr><tr><td></td><td>6</td><td>60</td><td>184</td><td>0.24</td><td>0.03</td></tr></tbody></table>

all systems. Most of the generated correspondences still involve properties where labels are equal, for instance: https://schema.org/distribution and http://www.w3.org/ns/dcat#distribution. In terms of F-measure, Matcha and LogMapLt have the best and similar performance. With respect to the pairs, a higher number of correspondences has been generated for the pairs involving DCAT-v3. LogMapLt and Matcha are the systems that are able to deal with a higher number of matching pairs.

In 2022, this track ran for the first time. Last year, similar to this year, only Matcha, LogMap and LogMapLt were able to generate non-empty alignments, with LogMapLt being able to generate a higher number of correspondences. In terms of precision, Matcha and LogMapLt had a higher precision in detriment of recall.
This task mostly deals with the properties of metadata schemes. Still, dealing with properties
is a challenging task.

4.13. Knowledge Graph

This year we evaluated all participants with the MELT framework to include all possible sub-
mission formats i.e. SEALS, and Web format. First, all systems are evaluated on a very small
matching task<fnref n="34" /> (even those not registered for the track). This revealed that not all systems were
able to handle the task, and in the end, 6 matchers can provide results for at least one test case.

Similar to the previous years, some systems like AMD need a post-processing step of the
resulting alignment file to be able to parse it. The reason is that the KGs in the knowledge graph
track contain special characters, e.g. ampersand. These characters need to be encoded in order to
parse these XML-formatted files correctly. The resulting alignments are available for download
<fnref n="35" />

Table 19 shows the results for all systems divided into class, property, instance, and overall
results. This also includes the number of tasks in which they were able to generate a non-empty
alignment (#tasks) and the average number of generated correspondences (size). We report the
macro averaged precision, F-measure, and recall results, where we do not distinguish empty
and erroneous (or not generated) alignments. The values in parentheses show the results when
considering only nonempty alignments.

This year's best overall system is the baseline using the alternative labels (0.84 F-measure).
The highest recall is again achieved by Matcha (0.84). It returns more correspondences than all
others (263,822.2 on average) but is only able to match instances in this track. Detailed results
for each test case can be found on the OAEI results page of the track<fnref n="36" />.

Property matches are still not created by all systems. LogMap, Matcha, and SOR-
BETMatcher do not return any of those mappings. One reason might be that the proper-
ties are typed as *rdf:Property* and not distinguished into *owl:ObjectProperty* or
*owl:DatatypeProperty*. OLaLa reaches the best score with 0.83 F-Measure.

Regarding runtime, Matcha (14:30:03) and LogMapLt (64:48:07) were the slowest systems. In
comparison to last year, the runtimes increased quite a lot and the systems should focus more
on scalable solutions. Besides the baselines (which need around 12 minutes for all test cases)
LogMap (00:56:43) and SORBETMatcher (00:21:53) were the fastest systems.

For further analysis of the results, we also provide an online dashboard<fnref n="37" /> generated with
MELT[65]. It allows us to inspect the results on a correspondence level. Due to the large amount
of these correspondences, it can take some time to load the full dashboard.

4.14. Pharmacogenomics

For this first year of the Pharmacogenomics track, 2 systems registered, namely LogMap and
Matcha. The evaluation was performed using the MELT framework. Unfortunately, none

<footnote n="34">³⁴http://oaei.ontologymatching.org/2019/results/knowledgegraph/small_test.zip</footnote>

<footnote n="35">³⁵http://oaei.ontologymatching.org/2023/results/knowledgegraph/knowledgegraph-alignments.zip</footnote>

<footnote n="36">³⁶http://oaei.ontologymatching.org/2023/results/knowledgegraph/index.html</footnote>

<footnote n="37">³⁷http://oaei.ontologymatching.org/2023/results/knowledgegraph/knowledge-graphDashboard.html</footnote>
<caption>**Table 19**</caption>

Knowledge Graph track results, divided into class, property, and instance performance. For matchers that were not capable of completing all tasks, the numbers in parentheses denote the performance when only averaging across tasks that were completed.

<table><thead><tr><th>System</th><th>Time</th><th>tracks</th><th>Size</th><th>Prec.</th><th>F-m.</th><th>Rec.</th></tr></thead><tbody><tr><td colspan="7">class performance</td></tr><tr><td>BaselineAltLabel</td><td>00:11:37</td><td>5</td><td>16.4</td><td>1.00 (1.00)</td><td>0.71 (0.71)</td><td>0.59 (0.59)</td></tr><tr><td>BaselineLabel</td><td>00:11:27</td><td>5</td><td>16.4</td><td>1.00 (1.00)</td><td>0.71 (0.71)</td><td>0.59 (0.59)</td></tr><tr><td>LogMap</td><td>00:56:43</td><td>5</td><td>19.4</td><td>0.93 (0.93)</td><td>0.80 (0.80)</td><td>0.71 (0.71)</td></tr><tr><td>LogMapLt</td><td>64:48:07</td><td>4</td><td>23.0</td><td>0.80 (1.00)</td><td>0.55 (0.69)</td><td>0.43 (0.54)</td></tr><tr><td>LSMatch</td><td>04:47:07</td><td>5</td><td>23.6</td><td>0.97 (0.97)</td><td>0.74 (0.74)</td><td>0.64 (0.64)</td></tr><tr><td>Matcha</td><td>14:30:03</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td>OLaLa</td><td>02:55:06</td><td>5</td><td>18.6</td><td>0.98 (0.98)</td><td>0.68 (0.68)</td><td>0.53 (0.53)</td></tr><tr><td>SORBETMatcher</td><td>00:21:53</td><td>5</td><td>22.4</td><td>0.93 (0.93)</td><td>0.80 (0.80)</td><td>0.73 (0.73)</td></tr><tr><td colspan="7">property performance</td></tr><tr><td>BaselineAltLabel</td><td>00:11:37</td><td>5</td><td>47.8</td><td>0.99 (0.99)</td><td>0.76 (0.76)</td><td>0.66 (0.66)</td></tr><tr><td>BaselineLabel</td><td>00:11:27</td><td>5</td><td>47.8</td><td>0.99 (0.99)</td><td>0.76 (0.76)</td><td>0.66 (0.66)</td></tr><tr><td>LogMap</td><td>00:56:43</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td>LogMapLt</td><td>64:48:07</td><td>4</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td>LSMatch</td><td>04:47:07</td><td>5</td><td>85.6</td><td>0.73 (0.73)</td><td>0.71 (0.71)</td><td>0.69 (0.69)</td></tr><tr><td>Matcha</td><td>14:30:03</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td>OLaLa</td><td>02:55:06</td><td>5</td><td>73.6</td><td>0.86 (0.86)</td><td>0.83 (0.83)</td><td>0.81 (0.81)</td></tr><tr><td>SORBETMatcher</td><td>00:21:53</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td colspan="7">instance performance</td></tr><tr><td>BaselineAltLabel</td><td>00:11:37</td><td>5</td><td>4674.8</td><td>0.89 (0.89)</td><td>0.84 (0.84)</td><td>0.80 (0.80)</td></tr><tr><td>BaselineLabel</td><td>00:11:27</td><td>5</td><td>3641.8</td><td>0.95 (0.95)</td><td>0.80 (0.80)</td><td>0.71 (0.71)</td></tr><tr><td>LogMap</td><td>00:56:43</td><td>5</td><td>4012.4</td><td>0.90 (0.90)</td><td>0.78 (0.78)</td><td>0.69 (0.69)</td></tr><tr><td>LogMapLt</td><td>64:48:07</td><td>4</td><td>6653.8</td><td>0.73 (0.91)</td><td>0.67 (0.84)</td><td>0.62 (0.78)</td></tr><tr><td>LSMatch</td><td>04:47:07</td><td>5</td><td>5872.2</td><td>0.66 (0.66)</td><td>0.59 (0.59)</td><td>0.60 (0.60)</td></tr><tr><td>Matcha</td><td>14:30:03</td><td>5</td><td>263822.2</td><td>0.55 (0.55)</td><td>0.63 (0.63)</td><td>0.86 (0.86)</td></tr><tr><td>OLaLa</td><td>02:55:06</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td>SORBETMatcher</td><td>00:21:53</td><td>5</td><td>0.0</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td><td>0.00 (0.00)</td></tr><tr><td colspan="7">overall performance</td></tr><tr><td>BaselineAltLabel</td><td>00:11:37</td><td>5</td><td>4739.0</td><td>0.89 (0.89)</td><td>0.84 (0.84)</td><td>0.80 (0.80)</td></tr><tr><td>BaselineLabel</td><td>00:11:27</td><td>5</td><td>3706.0</td><td>0.95 (0.95)</td><td>0.80 (0.80)</td><td>0.71 (0.71)</td></tr><tr><td>LogMap</td><td>00:56:43</td><td>5</td><td>4031.8</td><td>0.90 (0.90)</td><td>0.77 (0.77)</td><td>0.68 (0.68)</td></tr><tr><td>LogMapLt</td><td>64:48:07</td><td>4</td><td>6676.8</td><td>0.73 (0.92)</td><td>0.66 (0.83)</td><td>0.61 (0.76)</td></tr><tr><td>LSMatch</td><td>04:47:07</td><td>5</td><td>5981.4</td><td>0.66 (0.66)</td><td>0.60 (0.60)</td><td>0.61 (0.61)</td></tr><tr><td>Matcha</td><td>14:30:03</td><td>5</td><td>263822.2</td><td>0.55 (0.55)</td><td>0.62 (0.62)</td><td>0.84 (0.84)</td></tr><tr><td>OLaLa</td><td>02:55:06</td><td>5</td><td>92.2</td><td>0.88 (0.88)</td><td>0.03 (0.03)</td><td>0.02 (0.02)</td></tr><tr><td>SORBETMatcher</td><td>00:21:53</td><td>5</td><td>22.4</td><td>0.93 (0.93)</td><td>0.01 (0.01)</td><td>0.00 (0.00)</td></tr></tbody></table>

of these systems successfully produced alignments between reified *n*-ary tuples representing pharmacogenomic knowledge units. Some configurations of the two systems output some alignments between other entities (e.g., components of pharmacogenomic tuples) but not between the *n*-ary tuples themselves.
## 5. Conclusions and Lessons Learned

As in previous campaigns, in 2023, we witnessed a healthy mix of new and returning systems, with an imbalanced participation in the tracks.

The **schema matching** tracks gather the highest number of participants; however still little substantial progress in terms of the quality of the results or run time of top matching systems. As already reported last year, we observe a performance plateau being reached by existing strategies and algorithms. It is also true that established matching systems tend to focus more on new tracks and datasets than on improving their performance in long-standing tracks, whereas new systems typically struggle to compete with established ones.

According to the **Conference** track, there are more systems with the ability to match properties (7 in 2023 vs. 5 in 2022). Several ML-based systems used reference alignments for training to a certain extent (explained in their system papers). It has already happened in the past. It calls for a discussion and perhaps more ML-based tracks.

Since the creation of the **Material Sciences and Engineering** track, a large amount of new ontologies have been developed and utilized in various MSE applications. In contrast to the early development stages of this track, those ontologies are now easily accessible on the Matportal<fnref n="38" />. In the future, the MSE track should be updated with the currently most used top and mid-level MSE ontologies, which include the BWMD-mid<fnref n="39" />, the MSEO<fnref n="40" />, the PMDco<fnref n="41" />, the prov-o<fnref n="42" /> and IOF-mat<fnref n="43" />. In the OntoCommons-project<fnref n="44" /> alignments of frequently used ontologies of the MSE application area are produced and will be used to further improve the MSE benchmark based on the project results. Apart from also considering frequently used domain and application ontologies, also multi-ontology matching, knowledge graph matching, e.g. using the AluTrace-data<fnref n="45" />, and the usage of background knowledge should be considered in future OAEI campaigns.

With respect to the cross-lingual version of the Conference, the **Multifarm** track still attracts too few number of participants. Despite this fact, this year new participants came up with alternative strategies (i.e., deep learning) with respect to the last campaigns.

In the **Food** track, none of the evaluated matchers finds all reference correspondences correctly. LogMapLt stands out for its very fast computing speed. Matcha obtains the best results for the FNC application. The usage of background knowledge available in CIQUAL and SIREN ontologies in terms of food description based on FoodON concepts should be considered in future OAEI campaigns.

The **Bio-ML** track incorporated significant updates and attracted several new machine learning-based participants. However, the number of symbolic participants decreased. The best-performing systems are not consistent across tasks and settings, demonstrating the diversity of our datasets. It is also worth noting that SORBETMatcher is the only system can participate in both equivalence

<footnote n="38">³⁸https://matportal.org/</footnote>

<footnote n="39">³⁹https://matportal.org/ontologies/BWMD-MID</footnote>

<footnote n="40">⁴⁰https://matportal.org/ontologies/MSEO</footnote>

<footnote n="41">⁴¹https://github.com/materialdigital/core-ontology/</footnote>

<footnote n="42">⁴²https://www.ebi.ac.uk/ols/ontologies/prov</footnote>

<footnote n="43">⁴³https://industrialontologies.org/working-groups/the-material-science-and-engineering-mse-working-group-wg/</footnote>

<footnote n="44">⁴⁴https://ontocommons.eu</footnote>

<footnote n="45">⁴⁵https://github.com/Mat-O-Lab/AluTraceProject</footnote>
and subsumption matching.

In the **Biodiversity and Ecology** track, none of the systems was able to detect manual mappings created by domain experts and requiring biodiversity domain-specific knowledge. In this year's edition, we confirmed the inability of most systems to handle SKOS natively, as well as very large ontologies. Additionally, some systems did not perform well on the thesauri tasks because those contained concepts with similar URIs that were, in fact, completely different.

The **Interactive matching** track also witnessed a small number of participants. Two systems participated this year. This is puzzling considering that this track is based on the *Anatomy* and *Conference* test cases, and those tracks had 9 and 11 participants, respectively. The process of programmatically querying the Oracle class used to simulate user interactions is simple enough that it should not be a deterrent for participation, but perhaps we should look at facilitating the process further in future OAEI editions by providing implementation examples.

The **Complex matching** track tackles a challenge task that attracts too few number of participants. This year, no system was able to complete the task. As several sub-tracks have been discontinued, the track is limited to the conference domain. This track welcomes new organizers.

The **Crosswalks Data Schema Matching** track involves different schema formats and ways of representing schema properties. This opens the possibility of creating a dedicated task relying on other formats than OWL/RDF.

Automatic instance-matching benchmark generation algorithms have been gaining popularity, as evidenced by the fact that they are used in all three instance-matching tracks of this OAEI edition. One aspect that has not been addressed in such algorithms is that, if the transformation is too extreme, the correspondence may be unrealistic and impossible to detect even by humans. As such, we argue that *human-in-the-loop* techniques can be exploited to do a preventive quality-checking of generated correspondences and refine the set of correspondences included in the final reference alignment.

In the **Knowledge graph** track, the overall best scores are still unbeaten. Furthermore, the proportion of matchers not able to produce property alignments is high. This might change next year with new and improved systems.

In the **Common knowledge graphs** track, which challenges matching systems to map the schema of large-scale, automatically constructed, and cross-domain knowledge graphs. The number of participants is similar to last year, with a new system participating and former systems adapting their approaches to scale up to the task size. However, with some systems only being able to produce alignments when applied to smaller versions of the KG datasets, we still look forward to having more participants in the next OAEI campaign.

For the first year of the **Pharmacogenomics** track, participation was limited with only 2 systems registered. Unfortunately, none of the participating systems were able to output alignments between the targeted reified *n*-ary tuples. We will investigate whether this originates from the absence of labels for tuples and the only presence of structural information or if other aspects are detrimental (e.g., arity, background domain knowledge that must be considered to produce most alignments). These results highlight the interest in considering domain-specific problems to design new methods like [56, 66] or enrich existing ones. Recall that the track features different types of alignments between individuals, which is a specificity of the considered alignment task. This raises the question of whether such a granular matching setting could be generalized to other instance matching tasks. Since the alignment task in this track is structure-based, it is particularly
well-adapted to approaches relying on Graph Neural Networks that learn embeddings of nodes to align on the basis of their neighborhoods [66]. All these reasons motivate to propose again the track in the next editions of OAEI and adapt it to evaluate Machine Learning-based matchers. We hope that the growing awareness about this track and its specificity will attract additional systems.

Like in previous OAEI editions, most participants provided a description of their systems and their experience in the evaluation, in the form of OAEI system papers. These papers, like the present one, have not been peer-reviewed. However, they are full contributions to this evaluation exercise, reflecting the effort and insight of matching systems developers, and providing details about those systems and the algorithms they implement.

As each year, fruitful discussions at the Ontology Matching Workshop point out different directions for future improvements in OAEI. This year, with a higher number of systems relying on Large Language Models, there was a discussion on the specific requirements and alternative ways for gathering the alignments generated by such resource-consuming systems. It has also been highlighted the need to push the adoption of SSSOM [24] (this year MELT has incorporated the format but still few systems have adopted it), as a way for delivering richer alignments in terms of metadata and justifications [67]. As already mentioned before, there were also some interrogations on the stability reached in some (open)-schema matching tasks (in particular Anatomy and Conference tracks) as the performance has been quite stable for several years. This requires a further analysis of the difficult parts of the matching task. Last but not least, new tracks addressing more application/use-oriented tasks should be addressed and they are more than welcome.

The Ontology Alignment Evaluation Initiative will strive to remain a reference to the ontology matching community by improving both the test cases and the testing methodology to better reflect actual needs, as well as to promote progress in this field. More information can be found at: http://oaei.ontologymatching.org.

## Acknowledgments

We warmly thank the participants of this campaign. We know that they have worked hard to have their matching tools executable in time and they provided useful reports on their experience. The best way to learn about the results remains to read the papers that follow.

We are also grateful to Martin Ringwald and Terry Hayamizu for providing the reference alignment for the anatomy ontologies and thank Elena Beisswanger for her thorough support in improving the dataset's quality.

We also thank for their support, the past members of the Ontology Alignment Evaluation Initiative steering committee: Jérôme Euzenat (INRIA, FR), Yannis Kalfoglou (Ricoh laboratories, UK), Miklos Nagy (The Open University, UK), Natasha Noy (Google Inc., USA), Yuzhong Qu (Southeast University, CN), York Sure (Leibniz Gemeinschaft, DE), Jie Tang (Tsinghua University, CN), Heiner Stuckenschmidt (Mannheim Universität, DE), and George Vouros (University of the Aegean, GR).

Daniel Faria and Catia Pesquita were supported by the FCT through the LASIGE Research Unit (UIDB/00408/2020 and UIDP/00408/2020) and by the KATY project funded by the European Union's Horizon 2020 research and innovation program under grant agreement No 101017453.
Ernesto Jimenez-Ruiz has been partially supported by the SIRIUS Centre for Scalable Data Access (Research Council of Norway, project no.: 237889).

Irini Fundulaki and Tzanina Saveta were supported by the EU's Horizon 2020 research and innovation program under grant agreement No 688227 (Hobbit).

Patrick Lambrix, Huanyu Li, Mina Abd Nikooie Pour and Ying Li have been supported by the Swedish e-Science Research Centre (SeRC) and the Swedish National Graduate School in Computer Science (CUGS).

Beyza Yaman has been supported by ADAPT SFI Research Centre [grant 13/RC/2106_P2].

Jiaoyan Chen, Hang Dong, Yuan He, and Ian Horrocks have been supported by Samsung Research UK (SRUK) and the EPSRC project ConCur (EP/V050869/1).

Naouel Karam and Alsayed Algergawy have been supported by the German Research Foundation in the context of NFDI4BioDiversity project (number 442032008) and the CRC 1076 AquaDiva. We would like to thank Jessica Titocci, Martina Pulieri and Ilaria Rosati for providing the datasets for the biodiv SKOS thesauri tasks.

## References

[1] J. Euzenat, C. Meilicke, P. Shvaiko, H. Stuckenschmidt, C. Trojahn dos Santos, Ontology alignment evaluation initiative: six years of experience, Journal on Data Semantics XV (2011) 158–192.

[2] J. Euzenat, P. Shvaiko, Ontology matching, 2nd ed., Springer-Verlag, 2013.

[3] Y. Sure, O. Corcho, J. Euzenat, T. Hughes (Eds.), Proceedings of the Workshop on Evaluation of Ontology-based Tools (EON), Hiroshima (JP), 2004.

[4] B. Ashpole, M. Ehrig, J. Euzenat, H. Stuckenschmidt (Eds.), Proc. K-Cap Workshop on Integrating Ontologies, Banff (Canada), 2005. URL: http://ceur-ws.org/Vol-156/.

[5] M. Abd Nikooie Pour, A. Algergawy, P. Buche, L. J. Castro, J. Chen, H. Dong, O. Fallatah, D. Faria, I. Fundulaki, S. Hertling, Y. He, I. Horrocks, M. Huschka, L. Ibanescu, E. Jiménez-Ruiz, N. Karam, A. Laadhar, P. Lambrix, H. Li, Y. Li, F. Michel, E. Nasr, H. Paulheim, C. Pesquita, T. Saveta, P. Shvaiko, C. Trojahn, C. Verhey, M. Wu, B. Yaman, O. Zamazal, L. Zhou, Results of the ontology alignment evaluation initiative 2022, in: P. Shvaiko, J. Euzenat, E. Jiménez-Ruiz, O. Hassanzadeh, C. Trojahn (Eds.), Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) co-located with the 21st International Semantic Web Conference (ISWC 2022), Hangzhou, China, held as a virtual conference, October 23, 2022, volume 3324 of *CEUR Workshop Proceedings*, CEUR-WS.org, 2022, pp. 84–128. URL: https://ceur-ws.org/Vol-3324/oaei22_paper0.pdf.

[6] M. Abd Nikooie Pour, A. Algergawy, F. Amardeilh, R. Amini, O. Fallatah, D. Faria, I. Fundulaki, I. Harrow, S. Hertling, P. Hitzler, M. Huschka, L. Ibanescu, E. Jiménez-Ruiz, N. Karam, A. Laadhar, P. Lambrix, H. Li, Y. Li, F. Michel, E. Nasr, H. Paulheim, C. Pesquita, J. Portisch, C. Roussey, T. Saveta, P. Shvaiko, A. Splendiani, C. Trojahn, J. Vatascinová, B. Yaman, O. Zamazal, L. Zhou, Results of the ontology alignment evaluation initiative 2021, in: P. Shvaiko, J. Euzenat, E. Jiménez-Ruiz, O. Hassanzadeh, C. Trojahn (Eds.), Proceedings of the 16th International Workshop on Ontology Matching co-located with the 20th International Semantic Web Conference (ISWC 2021), Virtual conference, October 25,
2021, volume 3063 of *CEUR Workshop Proceedings*, CEUR-WS.org, 2021, pp. 62–108.
URL: http://ceur-ws.org/Vol-3063/oaei21_paper0.pdf.

[7] M. Abd Nikooie Pour, A. Algergawy, R. Amini, D. Faria, I. Fundulaki, I. Harrow, S. Hertling, E. Jiménez-Ruiz, C. Jonquet, N. Karam, A. Khiat, A. Laadhar, P. Lambrix, H. Li, Y. Li, P. Hitzler, H. Paulheim, C. Pesquita, T. Saveta, P. Shvaiko, A. Splendiani, É. Thiéblin, C. Trojahn, J. Vatascinová, B. Yaman, O. Zamazal, L. Zhou, Results of the ontology alignment evaluation initiative 2020, in: P. Shvaiko, J. Euzenat, E. Jiménez-Ruiz, O. Has-sanzadeh, C. Trojahn (Eds.), *Proceedings of the 15th International Workshop on Ontology Matching co-located with the 19th International Semantic Web Conference (ISWC 2020)*, Virtual conference (originally planned to be in Athens, Greece), November 2, 2020, vol-ume 2788 of *CEUR Workshop Proceedings*, CEUR-WS.org, 2020, pp. 92–138. URL: http://ceur-ws.org/Vol-2788/oaei20_paper0.pdf.

[8] A. Algergawy, D. Faria, A. Ferrara, I. Fundulaki, I. Harrow, S. Hertling, E. Jiménez-Ruiz, N. Karam, A. Khiat, P. Lambrix, H. Li, S. Montanelli, H. Paulheim, C. Pesquita, T. Saveta, P. Shvaiko, A. Splendiani, É. Thiéblin, C. Trojahn, J. Vatascinová, O. Zamazal, L. Zhou, Results of the ontology alignment evaluation initiative 2019, in: *Proceedings of the 14th International Workshop on Ontology Matching*, Auckland, New Zealand, 2019, pp. 46–85.

[9] A. Algergawy, M. Cheatham, D. Faria, A. Ferrara, I. Fundulaki, I. Harrow, S. Hertling, E. Jiménez-Ruiz, N. Karam, A. Khiat, P. Lambrix, H. Li, S. Montanelli, H. Paulheim, C. Pesquita, T. Saveta, D. Schmidt, P. Shvaiko, A. Splendiani, É. Thiéblin, C. Trojahn, J. Vatascinová, O. Zamazal, L. Zhou, Results of the ontology alignment evaluation initiative 2018, in: *Proceedings of the 13th International Workshop on Ontology Matching*, Monterey (CA, US), 2018, pp. 76–116.

[10] M. Achichi, M. Cheatham, Z. Dragisic, J. Euzenat, D. Faria, A. Ferrara, G. Flouris, I. Fundulaki, I. Harrow, V. Ivanova, E. Jiménez-Ruiz, K. Kolthoff, E. Kuss, P. Lambrix, H. Leopold, H. Li, C. Meilicke, M. Mohammadi, S. Montanelli, C. Pesquita, T. Saveta, P. Shvaiko, A. Splendiani, H. Stuckenschmidt, É. Thiéblin, K. Todorov, C. Trojahn, O. Zamazal, Results of the ontology alignment evaluation initiative 2017, in: *Proceedings of the 12th International Workshop on Ontology Matching*, Vienna, Austria, 2017, pp. 61–113. URL: http://ceur-ws.org/Vol-2032/oaei17_paper0.pdf.

[11] M. Achichi, M. Cheatham, Z. Dragisic, J. Euzenat, D. Faria, A. Ferrara, G. Flouris, I. Fundulaki, I. Harrow, V. Ivanova, E. Jiménez-Ruiz, E. Kuss, P. Lambrix, H. Leopold, H. Li, C. Meilicke, S. Montanelli, C. Pesquita, T. Saveta, P. Shvaiko, A. Splendiani, H. Stuckenschmidt, K. Todorov, C. Trojahn, O. Zamazal, Results of the ontology alignment evaluation initiative 2016, in: *Proceedings of the 11th International Ontology matching workshop*, Kobe (JP), 2016, pp. 73–129.

[12] M. Cheatham, Z. Dragisic, J. Euzenat, D. Faria, A. Ferrara, G. Flouris, I. Fundulaki, R. Granada, V. Ivanova, E. Jiménez-Ruiz, P. Lambrix, S. Montanelli, C. Pesquita, T. Saveta, P. Shvaiko, A. Solimando, C. Trojahn, O. Zamazal, Results of the ontology alignment evaluation initiative 2015, in: *Proceedings of the 10th International Ontology matching workshop*, Bethlehem (PA, US), 2015, pp. 60–115.

[13] Z. Dragisic, K. Eckert, J. Euzenat, D. Faria, A. Ferrara, R. Granada, V. Ivanova, E. Jiménez-Ruiz, A. O. Kempf, P. Lambrix, S. Montanelli, H. Paulheim, D. Ritze, P. Shvaiko, A. Solimando, C. T. dos Santos, O. Zamazal, B. C. Grau, Results of
the ontology alignment evaluation initiative 2014, in: Proceedings of the 9th International Ontology matching workshop, Riva del Garda (IT), 2014, pp. 61–104. URL: http://ceur-ws.org/Vol-1317/oaei14_paper0.pdf.

[14] B. Cuenca Grau, Z. Dragisic, K. Eckert, J. Euzenat, A. Ferrara, R. Granada, V. Ivanova, E. Jiménez-Ruiz, A. Kempf, P. Lambrix, A. Nikolov, H. Paulheim, D. Ritze, F. Scharffe, P. Shvaiko, C. Trojahn dos Santos, O. Zamazal, Results of the ontology alignment evaluation initiative 2013, in: P. Shvaiko, J. Euzenat, K. Srinivas, M. Mao, E. Jiménez-Ruiz (Eds.), Proceedings of the 8th International Ontology matching workshop, Sydney (NSW, AU), 2013, pp. 61–100. URL: http://oaei.ontologymatching.org/2013/results/oaei2013.pdf.

[15] J. Aguirre, B. Cuenca Grau, K. Eckert, J. Euzenat, A. Ferrara, R. van Hague, L. Hollink, E. Jiménez-Ruiz, C. Meilicke, A. Nikolov, D. Ritze, F. Scharffe, P. Shvaiko, O. Sváb-Zamazal, C. Trojahn, B. Zapilko, Results of the ontology alignment evaluation initiative 2012, in: Proceedings of the 7th International Ontology matching workshop, Boston (MA, US), 2012, pp. 73–115. URL: http://oaei.ontologymatching.org/2012/results/oaei2012.pdf.

[16] J. Euzenat, A. Ferrara, R. van Hague, L. Hollink, C. Meilicke, A. Nikolov, F. Scharffe, P. Shvaiko, H. Stuckenschmidt, O. Sváb-Zamazal, C. Trojahn dos Santos, Results of the ontology alignment evaluation initiative 2011, in: Proceedings of the 6th International Ontology matching workshop, Bonn (DE), 2011, pp. 85–110.

[17] J. Euzenat, A. Ferrara, C. Meilicke, A. Nikolov, J. Pane, F. Scharffe, P. Shvaiko, H. Stuckenschmidt, O. Sváb-Zamazal, V. Svátek, C. Trojahn dos Santos, Results of the ontology alignment evaluation initiative 2010, in: Proceedings of the 5th International Ontology matching workshop, Shanghai (CN), 2010, pp. 85–117. URL: http://oaei.ontologymatching.org/2010/results/oaei2010.pdf.

[18] J. Euzenat, A. Ferrara, L. Hollink, A. Isaac, C. Joslyn, V. Malaisé, C. Meilicke, A. Nikolov, J. Pane, M. Sabou, F. Scharffe, P. Shvaiko, V. Spiliopoulos, H. Stuckenschmidt, O. Sváb-Zamazal, V. Svátek, C. Trojahn dos Santos, G. Vouros, S. Wang, Results of the ontology alignment evaluation initiative 2009, in: Proceedings of the 4th International Ontology matching workshop, Chantilly (VA, US), 2009, pp. 73–126.

[19] C. Caracciolo, J. Euzenat, L. Hollink, R. Ichise, A. Isaac, V. Malaisé, C. Meilicke, J. Pane, P. Shvaiko, H. Stuckenschmidt, O. Sváb-Zamazal, V. Svátek, Results of the ontology alignment evaluation initiative 2008, in: Proceedings of the 3rd Ontology matching workshop, Karlsruhe (DE), 2008, pp. 73–120.

[20] J. Euzenat, A. Isaac, C. Meilicke, P. Shvaiko, H. Stuckenschmidt, O. Svab, V. Svatek, W. van Hage, M. Yatskevich, Results of the ontology alignment evaluation initiative 2007, in: Proceedings 2nd International Ontology matching workshop, Busan (KR), 2007, pp. 96–132. URL: http://ceur-ws.org/Vol-304/paper9.pdf.

[21] J. Euzenat, M. Mochol, P. Shvaiko, H. Stuckenschmidt, O. Svab, V. Svatek, W. R. van Hage, M. Yatskevich, Results of the ontology alignment evaluation initiative 2006, in: Proceedings of the 1st International Ontology matching workshop, Athens (GA, US), 2006, pp. 73–95. URL: http://ceur-ws.org/Vol-225/paper7.pdf.

[22] S. Hertling, J. Portisch, H. Paulheim, Melt - matching evaluation toolkit, in: M. Acosta, P. Cudré-Mauroux, M. Maleshkova, T. Pellegrini, H. Sack, Y. Sure-Vetter (Eds.), Semantic Systems. The Power of AI and Knowledge Graphs, Springer International Publishing, Cham, 2019, pp. 231–245.
[23] E. Jiménez-Ruiz, T. Saveta, O. Zamazal, S. Hertling, M. Röder, I. Fundulaki, A.-C. N. Ngomo, M. A. Sherif, A. Annane, Z. Bellahsene, S. B. Yahia, G. Diallo, D. Faria, M. Kachroudi, A. Khiat, P. Lambrix, H. Li, M. Mackeprang, M. Mohammadi, M. Rybinski, B. S. Balasubramani, C. Trojahn, Introducing the HOBBIT platform into the Ontology Alignment Evaluation Campaign, in: Proceedings of the 13th International Workshop on Ontology Matching, 2018.

[24] N. Matentzoglu, J. P. Balhoff, S. M. Bello, C. Bizon, M. Brush, T. J. Callahan, C. G. Chute, W. D. Duncan, C. T. Evelo, D. Gabriel, J. Graybeal, A. Gray, B. M. Gyori, M. Haendel, H. Harmse, N. L. Harris, I. Harrow, H. B. Hegde, A. L. Hoyt, C. T. Hoyt, D. Jiao, E. Jiménez-Ruiz, S. Jupp, H. Kim, S. Koehler, T. Liener, Q. Long, J. Malone, J. A. McLaughlin, J. A. McMurry, S. Moxon, M. C. Munoz-Torres, D. Osumi-Sutherland, J. A. Overton, B. Peters, T. Putman, N. Queralt-Rosinach, K. Shefchek, H. Solbrig, A. Thessen, T. Tudorache, N. Vasilevsky, A. H. Wagner, C. J. Mungall, A Simple Standard for Sharing Ontological Mappings (SSSOM), Database 2022 (2022) baac035. URL: https://doi.org/10.1093/database/baac035. doi:10.1093/database/baac035.

[25] Z. Dragisic, V. Ivanova, H. Li, P. Lambrix, Experiences from the anatomy track in the ontology alignment evaluation initiative, Journal of Biomedical Semantics 8 (2017) 56:1–56:28. URL: https://doi.org/10.1186/s13326-017-0166-5. doi:10.1186/s13326-017-0166-5.

[26] O. Zamazal, V. Svátek, The ten-year ontofarm and its fertilization within the onto-sphere, Web Semantics: Science, Services and Agents on the World Wide Web 43 (2017) 46–53.

[27] C. Meilicke, R. García Castro, F. Freitas, W. van Hage, E. Montiel-Ponsoda, R. Ribeiro de Azevedo, H. Stuckenschmidt, O. Sváb-Zamazal, V. Svátek, A. Tamilin, C. Trojahn, S. Wang, MultiFarm: A benchmark for multilingual ontology matching, Journal of web semantics 15 (2012) 62–68. URL: http://www.sciencedirect.com/science/article/pii/S157082681200039X. doi:10.1016/j.websem.2012.04.001.

[28] P. Buche, J. Cufi, S. Dervaux, J. Dibie, L. Ibanescu, A. Oudot, M. Weber, How to manage incompleteness of nutritional food sources?: A solution using foodon as pivot ontology, Int. J. Agric. Environ. Inf. Syst. 12 (2021) 1–26. URL: https://doi.org/10.4018/ijaeis.20211001.oa4. doi:10.4018/ijaeis.20211001.oa4.

[29] H. Paulheim, S. Hertling, D. Ritze, Towards evaluating interactive ontology matching tools, in: Proceedings of the 10th Extended Semantic Web Conference, Montpellier (FR), 2013, pp. 31–45. URL: http://dx.doi.org/10.1007/978-3-642-38288-8_3.

[30] Z. Dragisic, V. Ivanova, P. Lambrix, D. Faria, E. Jiménez-Ruiz, C. Pesquita, User validation in ontology alignment, in: Proceedings of the 15th International Semantic Web Conference, Kobe (JP), 2016, pp. 200–217. URL: http://dx.doi.org/10.1007/978-3-319-46523-4_13. doi:10.1007/978-3-319-46523-4_13.

[31] H. Li, Z. Dragisic, D. Faria, V. Ivanova, E. Jiménez-Ruiz, P. Lambrix, C. Pesquita, User validation in ontology alignment: functional assessment and impact, The Knowledge Engineering Review 34 (2019) e15. doi:10.1017/S0269888919000080.

[32] V. Ivanova, P. Lambrix, J. Åberg, Requirements for and evaluation of user support for large-scale ontology alignment, in: Proceedings of the European Semantic Web Conference, 2015, pp. 3–20.

[33] Y. He, J. Chen, H. Dong, E. Jiménez-Ruiz, A. Hadian, I. Horrocks, Machine learning-
friendly biomedical datasets for equivalence and subsumption ontology matching, in: U. Sattler, A. Hogan, C. M. Keet, V. Presutti, J. P. A. Almeida, H. Takeda, P. Monnin, G. Pirrò, C. d'Amato (Eds.), The Semantic Web - ISWC 2022 - 21st International Semantic Web Conference, Virtual Event, October 23-27, 2022, Proceedings, volume 13489 of Lecture Notes in Computer Science, Springer, 2022, pp. 575-591. URL: https://doi.org/10.1007/978-3-031-19433-7_33. doi:10.1007/978-3-031-19433-7_33.

[34] N. A. Vasilevsky, N. A. Matentzoglu, S. Toro, J. E. Flack IV, H. Hegde, D. R. Unni, G. F. Alyea, J. S. Amberger, L. Babb, J. P. Balhoff, et al., Mondo: Unifying diseases for the world, by the world, medRxiv (2022) 2022-04.

[35] O. Bodenreider, The unified medical language system (umls): integrating biomedical terminology, Nucleic acids research (2004).

[36] E. Jiménez-Ruiz, B. C. Grau, LogMap: Logic-based and scalable ontology matching, in: Proceedings of the 10th International Semantic Web Conference, Bonn (DE), 2011, pp. 273-288.

[37] Y. He, J. Chen, H. Dong, I. Horrocks, Exploring large language models for ontology alignment, arXiv preprint arXiv:2309.07172 (2023).

[38] Y. He, J. Chen, H. Dong, I. Horrocks, C. Allocca, T. Kim, B. Sapkota, Deeponto: A python package for ontology engineering with deep learning, arXiv preprint arXiv:2307.03067 (2023).

[39] N. Karam, C. Müller-Birn, M. Gleisberg, D. Fichtmüller, R. Tolksdorf, A. Güntsch, A terminology service supporting semantic annotation, integration, discovery and analysis of interdisciplinary research data, Datenbank-Spektrum 16 (2016) 195-205. URL: https://doi.org/10.1007/s13222-016-0231-8. doi:10.1007/s13222-016-0231-8.

[40] F. Klan, E. Faessler, A. Algergawy, B. König-Ries, U. Hahn, Integrated semantic search on structured and unstructured data in the adonis system, in: Proceedings of the 2nd International Workshop on Semantics for Biodiversity, 2017.

[41] N. Karam, A. Khiat, A. Algergawy, M. Sattler, C. Weiland, M. Schmidt, Matching biodiversity and ecology ontologies: challenges and evaluation results, Knowl. Eng. Rev. 35 (2020) e9. URL: https://doi.org/10.1017/S0269888920000132. doi:10.1017/S0269888920000132.

[42] F. Michel, O. Gargominy, S. Tercerie, C. Faron-Zucker, A Model to Represent Nomenclatural and Taxonomic Information as Linked Data. Application to the French Taxonomic Register, TAXREF, in: A. Algergawy, N. Karam, F. Klan, C. Jonquet (Eds.), Proceedings of the 2nd International Workshop on Semantics for Biodiversity co-located with 16th International Semantic Web Conference (ISWC 2017), Vienna, Austria, October 22nd, 2017, volume 1933 of CEUR Workshop Proceedings, CEUR-WS.org, 2017.

[43] A. Algergawy, N. Karam, A. Laadhar, F. Michel, Too big to match: a strategy around matching tasks for large taxonomies, in: P. Shvaiko, J. Euzenat, E. Jiménez-Ruiz, O. Hassanzadeh, C. Trojahn (Eds.), Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) co-located with the 21st International Semantic Web Conference (ISWC 2022), Hangzhou, China, held as a virtual conference, October 23, 2022, volume 3324 of CEUR Workshop Proceedings, CEUR-WS.org, 2022, pp. 67-72. URL: https://ceur-ws.org/Vol-3324/om2022_STpaper1.pdf.

[44] T. Ashino, Materials Ontology: An Infrastructure for Exchanging Materials Information
and Knowledge, Data Science Journal 9 (2010) 54–61.

[45] M. Wu, P. Hagan, B. Cecconi, S. M. Richard, C. Verhey, R. R. M. S. WG, A collection of crosswalks from fifteen research data schemas to schema.org, 2022. URL: https://doi.org/10.15497/RDA00069. doi:10.15497/RDA00069.

[46] M. Wu, S. M. Richard, C. Verhey, L. J. Castro, B. Cecconi, N. Juty, An analysis of crosswalks from research data schemas to schema.org, Data Intelligence (2022) 1–21. URL: https://doi.org/10.1162/dint_a_00186. doi:10.1162/dint_a_00186.

[47] C. Bizer, J. Lehmann, R. Isele, M. Jakob, A. Jentzsch, D. Kontokostas, P. N. Mende, S. Hellmann, M. Morsey, P. van Kleef, S. Auer, C. Bizer, DBpedia – A Large-scale, Multilingual Knowledge Base Extracted from Wikipedia, Semantic Web (2012) 1–5.

[48] F. M. Suchanek, G. Kasneci, G. Weikum, Yago: a core of semantic knowledge, in: Proceedings of the 16th International Conference on World Wide Web, 2007, pp. 697–706.

[49] A. Carlson, J. Betteridge, B. Kisiel, B. Settles, E. R. Hruschka, T. M. Mitchell, Toward an architecture for never-ending language learning, in: Twenty-Fourth AAAI Conference on AI, 2010.

[50] O. Fallatah, Z. Zhang, F. Hopfgartner, The impact of imbalanced class distribution on knowledge graphs matching, in: Proceedings of the 17th International Workshop on Ontology Matching (OM 2022). CEUR-WS, 2022.

[51] O. Fallatah, Z. Zhang, F. Hopfgartner, A gold standard dataset for large knowledge graphs matching, in: Ontology Matching 2020: Proceedings of the 15th International Workshop on Ontology Matching co-located with (ISWC 2020), 2020.

[52] P. Krauss, schemaorg-wikidata-map, https://github.com/okfn-brasil/schemaOrg-Wikidata-Map, 2017.

[53] S. Hertling, H. Paulheim, Dbkwik: extracting and integrating knowledge from thousands of wikis, Knowledge and Information Systems (2019).

[54] S. Hertling, H. Paulheim, Dbkwik: A consolidated knowledge graph from thousands of wikis, in: Proceedings of the International Conference on Big Knowledge, 2018.

[55] P. Monnin, A. Coulet, Matching pharmacogenomic knowledge: particularities, results, and perspectives, in: P. Shvaiko, J. Euzenat, E. Jiménez-Ruiz, O. Hassanzadeh, C. Trojahn (Eds.), Proceedings of the 17th International Workshop on Ontology Matching (OM 2022) co-located with the 21th International Semantic Web Conference (ISWC 2022), Hangzhou, China, held as a virtual conference, October 23, 2022, volume 3324 of *CEUR Workshop Proceedings*, CEUR-WS.org, 2022, pp. 79–83. URL: https://ceur-ws.org/Vol-3324/om2022_STpaper3.pdf.

[56] P. Monnin, M. Couceiro, A. Napoli, A. Coulet, Knowledge-based matching of n-ary tuples, in: M. Alam, T. Braun, B. Yun (Eds.), Ontologies and Concepts in Mind and Machine - 25th International Conference on Conceptual Structures, ICCS 2020, Bolzano, Italy, September 18-20, 2020, Proceedings, volume 12277 of *Lecture Notes in Computer Science*, Springer, 2020, pp. 48–56. URL: https://doi.org/10.1007/978-3-030-57855-8_4. doi:10.1007/978-3-030-57855-8_4.

[57] P. Monnin, J. Legrand, G. Husson, P. Ringot, A. Tchechmedjiev, C. Jonquet, A. Napoli, A. Coulet, PGxO and PGxLOD: a reconciliation of pharmacogenomic knowledge of various provenances, enabling further comparison, BMC Bioinformatics 20-S (2019) 139:1–139:16. URL: https://doi.org/10.1186/s12859-019-2693-9. doi:10.1186/
S12859-019-2693-9.

[58] Y. He, J. Chen, D. Antonyrajah, I. Horrocks, Bertmap: a bert-based ontology alignment system, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, 2022, pp. 5684–5691.

[59] J. Chen, Y. He, Y. Geng, E. Jiménez-Ruiz, H. Dong, I. Horrocks, Contextual semantic embeddings for ontology subsumption prediction, World Wide Web (2023) 1–23.

[60] J. Dabrowski, E. V. Munson, 40 years of searching for the best computer system response time, Interacting with Computers 23 (2011) 555–564. URL: http://www.sciencedirect.com/science/article/pii/S0953543811000579. doi:http://dx.doi.org/10.1016/j.intcom.2011.05.008.

[61] Z. Wang, I. F. Cruz, Agreementmakerdeep results for oaei 2021., in: OM@ ISWC, 2021, pp. 124–130.

[62] D. Faria, M. C. Silva, P. Cotovio, P. Eugénio, C. Pesquita, Matcha and matcha-dl results for oaei 2022, 2022.

[63] F. Gosselin, A. Zouaq, Sorbet: A siamese network for ontology embeddings using a distance-based regression loss and bert, in: International Semantic Web Conference, Springer, 2023, pp. 561–578.

[64] J. Chen, P. Hu, E. Jimenez-Ruiz, O. M. Holter, D. Antonyrajah, I. Horrocks, Owl2vec*: Embedding of owl ontologies, Machine Learning 110 (2021) 1813–1845.

[65] J. Portisch, S. Hertling, H. Paulheim, Visual analysis of ontology matching results with the melt dashboard, in: European Semantic Web Conference, 2020, pp. 186–190.

[66] P. Monnin, C. Raïssi, A. Napoli, A. Coulet, Discovering alignment relations with graph convolutional networks: A biomedical case study, Semantic Web 13 (2022) 379–398. URL: https://doi.org/10.3233/SW-210452. doi:10.3233/SW-210452.

[67] N. Matentzoglu, I. Braun, A. R. Caron, D. Goutte-Gattat, B. M. Gyori, N. L. Harris, E. Hartley, H. B. Hegde, S. Hertling, C. Tapley, H. Kim, H. Li, J. McLaughlin, C. Trojahn, N. Vasilevsky, C. Mungall, A Simple Standard for Ontological Mappings 2023: Updates on data model, collaborations and tooling, in: OM 2023: The 18th International Workshop on Ontology Matching collocated with the 22nd International Semantic Web Conference ISWC-2023 November 7th, 2023, Athens, Greece, 2023.