# Uncertainty Management in the Construction of Knowledge Graphs: a Survey

Miguel Couceiro, Lucas Jarnac, Yoan Chabot

<figure><img src="image_1.png" /></figure>

## To cite this version:

Miguel Couceiro, Lucas Jarnac, Yoan Chabot. Uncertainty Management in the Construction of Knowledge Graphs: a Survey. 2024. hal-04596656

HAL Id: hal-04596656

https://inria.hal.science/hal-04596656

Preprint submitted on 31 May 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Uncertainty Management in the Construction of Knowledge Graphs: a Survey

Lucas Jarnac, Yoan Chabot, Miguel Couceiro

► To cite this version:

Lucas Jarnac, Yoan Chabot, Miguel Couceiro. Uncertainty Management in the Construction of Knowledge Graphs: a Survey. Transactions on Graph Data and Knowledge, 2025, 3 (1), pp.1-48.
10.48550/arXiv.2405.16929 . hal-04596656

HAL Id: hal-04596656

https://inria.hal.science/hal-04596656v1

Submitted on 31 May 2024

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Uncertainty Management in the Construction
of Knowledge Graphs: a Survey

Lucas Jarnac<sup>a,b,*</sup>, Yoan Chabot<sup>a</sup> and Miguel Couceiro<sup>b,c</sup>

<sup>a</sup> Orange, France
E-mails: lucas.jarnac@orange.com, yoan.chabot@orange.com

<sup>b</sup> Université de Lorraine, CNRS, LORIA, Nancy, France

E-mail: miguel.couceiro@loria.fr

<sup>c</sup> INESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal

**Abstract.** Knowledge Graphs (KGs) are a major asset for companies thanks to their great flexibility in data representation and their numerous applications, e.g., vocabulary sharing, Q/A or recommendation systems. To build a KG it is a common practice to rely on automatic methods for extracting knowledge from various heterogeneous sources. But in a noisy and uncertain world, knowledge may not be reliable and conflicts between data sources may occur. Integrating unreliable data would directly impact the use of the KG, therefore such conflicts must be resolved. This could be done manually by selecting the best data to integrate. This first approach is highly accurate, but costly and time-consuming. That is why recent efforts focus on automatic approaches, which represents a challenging task since it requires handling the uncertainty of extracted knowledge throughout its integration into the KG. We survey state-of-the-art approaches in this direction and present constructions of both open and enterprise KGs and how their quality is maintained. We then describe different knowledge extraction methods, introducing additional uncertainty. We also discuss downstream tasks after knowledge acquisition, including KG completion using embedding models, knowledge alignment, and knowledge fusion in order to address the problem of knowledge uncertainty in KG construction. We conclude with a discussion on the remaining challenges and perspectives when constructing a KG taking into account uncertainty.

Keywords: Knowledge reconciliation, Uncertainty, Heterogeneous sources, Knowledge graph construction

# 1. Introduction

Huge amounts of data expressed in the form of tables, texts, or databases are generated by organizations every day. When using these data within an organization, we have to deal with uncertainty, as the data often suffer from contradictions and differences in specificity. These are the effect of incompleteness, vagueness, fuzziness, invalidity, ambiguity, and timeliness leading to uncertainty about the correctness of the data [28]. The uncertainty can be due to the source of the data (e.g., a document written by an expert vs. a non-expert in the field concerned) or in the data itself (e.g., a scientific supposition where the fact is not yet well-defined but accepted by consensus as it is). For example, on the French Wikipedia page of the former president of France Jacques Chirac<fnref n="1" />, we can read that he was the mayor of Paris from March 25, 1977 to May 16, 1995 while on Wikidata<fnref n="2" /> it is mentioned that he was the mayor of Paris from March 20, 1977 to May 16, 1995 as depicted in Figure 1. In addition, data are not settled over time [118]. Some facts are known to change including all facts that involve a period of time for which the fact is

<footnote>*Corresponding author. E-mail: lucas.jarnac@orange.com.</footnote>

<footnote n="1">¹https://fr.wikipedia.org/wiki/Jacques_Chirac</footnote>

<footnote n="2">²https://www.wikidata.org/wiki/Q2105</footnote>
# Uncertainty Management in the Construction of Knowledge Graphs: a Survey

Lucas Jarnac<sup>a,b,*</sup>, Yoan Chabot<sup>a</sup> and Miguel Couceiro<sup>b,c</sup>

<sup>a</sup> Orange, France
E-mails: lucas.jarnac@orange.com, yoan.chabot@orange.com

<sup>b</sup> Université de Lorraine, CNRS, LORIA, Nancy, France

E-mail: miguel.couceiro@loria.fr

<sup>c</sup> INESC-ID, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal

**Abstract.** Knowledge Graphs (KGs) are a major asset for companies thanks to their great flexibility in data representation and their numerous applications, e.g., vocabulary sharing, Q/A or recommendation systems. To build a KG it is a common practice to rely on automatic methods for extracting knowledge from various heterogeneous sources. But in a noisy and uncertain world, knowledge may not be reliable and conflicts between data sources may occur. Integrating unreliable data would directly impact the use of the KG, therefore such conflicts must be resolved. This could be done manually by selecting the best data to integrate. This first approach is highly accurate, but costly and time-consuming. That is why recent efforts focus on automatic approaches, which represents a challenging task since it requires handling the uncertainty of extracted knowledge throughout its integration into the KG. We survey state-of-the-art approaches in this direction and present constructions of both open and enterprise KGs and how their quality is maintained. We then describe different knowledge extraction methods, introducing additional uncertainty. We also discuss downstream tasks after knowledge acquisition, including KG completion using embedding models, knowledge alignment, and knowledge fusion in order to address the problem of knowledge uncertainty in KG construction. We conclude with a discussion on the remaining challenges and perspectives when constructing a KG taking into account uncertainty.

**Keywords:** Knowledge reconciliation, Uncertainty, Heterogeneous sources, Knowledge graph construction

## 1. Introduction

Huge amounts of data expressed in the form of tables, texts, or databases are generated by organizations every day. When using these data within an organization, we have to deal with uncertainty, as the data often suffer from contradictions and differences in specificity. These are the effect of incompleteness, vagueness, fuzziness, invalidity, ambiguity, and timeliness leading to uncertainty about the correctness of the data [28]. The uncertainty can be due to the source of the data (e.g., a document written by an expert vs. a non-expert in the field concerned) or in the data itself (e.g., a scientific supposition where the fact is not yet well-defined but accepted by consensus as it is). For example, on the French Wikipedia page of the former president of France Jacques Chirac<fnref n="1" />, we can read that he was the mayor of Paris from March 25, 1977 to May 16, 1995 while on Wikidata<fnref n="2" /> it is mentioned that he was the mayor of Paris from March 20, 1977 to May 16, 1995 as depicted in Figure 1. In addition, data are not settled over time [118]. Some facts are known to change including all facts that involve a period of time for which the fact is

<footnote>*Corresponding author. E-mail: lucas.jarnac@orange.com.</footnote>

<footnote n="1">¹https://fr.wikipedia.org/wiki/Jacques_Chirac</footnote>

<footnote n="2">²https://www.wikidata.org/wiki/Q2105</footnote>
<figure><img src="image_3.png" /><figcaption>(a) French Wikipedia page of Jacques Chirac.</figcaption></figure>

<figure><img src="image_4.png" /></figure>

mayor of Paris

start time

end time

20 March 1977

16 May 1995

<caption>(b) Wikidata page of Jacques Chirac.</caption>

Fig. 1. Illustration of a contradiction between two sources. One of the sources claims that the mandate of Jacques Chirac as mayor of Paris began on March 25 while the other claims that it began on March 20.

valid (e.g., the mandate of a president or the place of residence of a person) or knowledge in specific domains may be known to change regularly, which is often the case for History where excavations can modify knowledge which, until today, was considered established.

Many knowledge graphs (KGs) have been built to represent such data in recent years, and they have become major asset for organizations, since KGs can support various downstream tasks such as knowledge and vocabulary indexation, as well as other applications in recommendation systems, question/answering systems, knowledge management, or search engine systems [61, 109]. To build or enrich a KG and reconcile uncertain data, we can rely on manual approaches (e.g., domain experts) but this is a time-consuming and tedious process. Alternatively, it is common to leverage automatic knowledge extraction approaches that handle large volumes of data from various heterogeneous sources, e.g., texts [35], tables [90], or databases to ensure the coverage of the KG. These automatic approaches are usually based on three main steps:

1. Extraction of knowledge from documents.

2. Detection of duplicates and conflicts between extracted knowledge. Conflicts occur from differences of specificity or knowledge contradictions.

3. Fusion of aligned knowledge: once detection is completed, conflicting knowledge should be reconciled.

However, each of the aforementioned steps is error-prone and increases the uncertainty on extracted knowledge due to the performance of the algorithms [72, 83, 146].

Uncertainty may also be found in knowledge, which we can distinguish two types: *objective knowledge* in which a single value is accepted (e.g., the mandate of Jacques Chirac where only one period of time is the true value), and *subjective knowledge* in which several values can be accepted according to their context and point of view (e.g., the number of participants in a protest depending on the counting technique). Most KG construction methods do not take into account noisy facts and the uncertainty inherent in extraction algorithms and knowledge, which may impact downstream applications. Therefore, there is a need to reconcile knowledge units extracted from heterogeneous sources before integrating them into the KG in order to obtain a single or multiple representations that are as reliable and fair as possible [109]. In this survey, we review different approaches to integrate knowledge uncertainty in the main steps of KG construction with up-to-date knowledge fusion methods and its representation in the graph. [114] surveys approaches and evaluation methods for KG refinement, particularly KG completion and error detection methods. In [87], the authors review truth discovery methods used in knowledge fusion before 2015. However, to the best of our knowledge, there is no survey about uncertainty handling in the construction of KGs.

The remainder of this survey is structured as follows. In Section 2, we describe our research methodology which led us to write this survey. We introduce the definition of KGs with some well-known KGs that have been built in recent years, then tools and quality metrics considered in KGs construction in Section 3. We present some knowledge extraction approaches and why they lead to uncertain knowledge in Section 4. An ideal knowledge integration pipeline handling the uncertainty of knowledge is provided in Section 5, while the steps of knowledge refinement from the pipeline are described in Section 6 for uncertainty consideration in KG representation learning, in Section 7 for knowledge alignment, and in Section 8 for knowledge fusion. The solutions for uncertainty representation in KGs are then listed and depicted in Section 9. We also discuss some perspectives on the use of uncertainty in the KG ecosystem in Section 10, before concluding this survey in Section 11.
<figure><img src="image_2.png" /><figcaption>(a) French Wikipedia page of Jacques Chirac.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) Wikidata page of Jacques Chirac.</figcaption></figure>

Fig. 1. Illustration of a contradiction between two sources. One of the sources claims that the mandate of Jacques Chirac as mayor of Paris began on March 25 while the other claims that it began on March 20.

valid (e.g., the mandate of a president or the place of residence of a person) or knowledge in specific domains may be known to change regularly, which is often the case for History where excavations can modify knowledge which, until today, was considered established.

Many knowledge graphs (KGs) have been built to represent such data in recent years, and they have become major asset for organizations, since KGs can support various downstream tasks such as knowledge and vocabulary indexation, as well as other applications in recommendation systems, question/answering systems, knowledge management, or search engine systems [61, 109]. To build or enrich a KG and reconcile uncertain data, we can rely on manual approaches (e.g., domain experts) but this is a time-consuming and tedious process. Alternatively, it is common to leverage automatic knowledge extraction approaches that handle large volumes of data from various heterogeneous sources, e.g., texts [35], tables [90], or databases to ensure the coverage of the KG. These automatic approaches are usually based on three main steps:

1. Extraction of knowledge from documents.

2. Detection of duplicates and conflicts between extracted knowledge. Conflicts occur from differences of specificity or knowledge contradictions.

3. Fusion of aligned knowledge: once detection is completed, conflicting knowledge should be reconciled.

However, each of the aforementioned steps is error-prone and increases the uncertainty on extracted knowledge due to the performance of the algorithms [72, 83, 146].

Uncertainty may also be found in knowledge, which we can distinguish two types: *objective knowledge* in which a single value is accepted (e.g., the mandate of Jacques Chirac where only one period of time is the true value), and *subjective knowledge* in which several values can be accepted according to their context and point of view (e.g., the number of participants in a protest depending on the counting technique). Most KG construction methods do not take into account noisy facts and the uncertainty inherent in extraction algorithms and knowledge, which may impact downstream applications. Therefore, there is a need to reconcile knowledge units extracted from heterogeneous sources before integrating them into the KG in order to obtain a single or multiple representations that are as reliable and fair as possible [109]. In this survey, we review different approaches to integrate knowledge uncertainty in the main steps of KG construction with up-to-date knowledge fusion methods and its representation in the graph. [114] surveys approaches and evaluation methods for KG refinement, particularly KG completion and error detection methods. In [87], the authors review truth discovery methods used in knowledge fusion before 2015. However, to the best of our knowledge, there is no survey about uncertainty handling in the construction of KGs.

The remainder of this survey is structured as follows. In Section 2, we describe our research methodology which led us to write this survey. We introduce the definition of KGs with some well-known KGs that have been built in recent years, then tools and quality metrics considered in KGs construction in Section 3. We present some knowledge extraction approaches and why they lead to uncertain knowledge in Section 4. An ideal knowledge integration pipeline handling the uncertainty of knowledge is provided in Section 5, while the steps of knowledge refinement from the pipeline are described in Section 6 for uncertainty consideration in KG representation learning, in Section 7 for knowledge alignment, and in Section 8 for knowledge fusion. The solutions for uncertainty representation in KGs are then listed and depicted in Section 9. We also discuss some perspectives on the use of uncertainty in the KG ecosystem in Section 10, before concluding this survey in Section 11.
## 2. Research Methodology

This paper aims at surveying methods to construct a KG from uncertain knowledge. In this section, we provide our research methodology for discovering and selecting papers on this purpose. To find papers of interest, we mainly used the Google Scholar search engine and created alerts with the following keywords: KG fusion, multi-source knowledge fusion, KG resolution, KG quality, knowledge fusion, KG reconciliation, KG alignment, KG matching, KG resolution, KG cleaning. The aforementioned keywords have been combined with the keyword “uncertain” and the terms “knowledge”, “data”, and “information” used interchangeably, e.g., “uncertain knowledge fusion”, “uncertain data fusion”, or “uncertain information fusion”.

For the selection of the papers, we proceeded as follows: (1) we look at the title of the paper, if it is relevant and seems to be related to one of the topics we were looking for, then we read the abstract; (2) if the paper presents a method related to uncertain KG construction or a method that is not related to KGs but which could be extrapolated to a KG, we selected it. We provide Figure 2 that depicts the distribution of paper publication years according to four applications of KG refinement: uncertain embedding, knowledge fusion, knowledge alignment, and uncertainty representation. We observe that the representation of uncertainty has been addressed in the literature since 2004, while the vector space representation of uncertainty is a more recent research topic (since 2016). The number of papers on knowledge fusion mentioned in this survey has remained constant over the last few years. This is due to new models based on deep learning that are being explored to tackle these tasks.

<figure><img src="image_5.png" /><figcaption>Fig. 2. Distribution of paper publication years according to uncertain KG embedding ( ), knowledge fusion ( ), knowledge alignment ( ), uncertainty representation ( ).</figcaption></figure>

## 3. Knowledge Graphs

Before going into further detail on reconciliation approaches, it is important to define KGs, which are the core of this survey. KGs provide a structural representation of knowledge that is captured by the relations between entities in the graph. The KGs provide a concise and intuitive data representation and abstraction, making them an ideal tool to manage knowledge of organizations in a sustainable way or to support search and querying applications [60], which led several companies to build their own KGs [109].

In this section, we provide a definition of KG, and we describe some well-known KGs including open KGs and Enterprise Knowledge Graphs (EKGs) and how their consistency is maintained in Section 3.1.
## 2. Research Methodology

This paper aims at surveying methods to construct a KG from uncertain knowledge. In this section, we provide our research methodology for discovering and selecting papers on this purpose. To find papers of interest, we mainly used the Google Scholar search engine and created alerts with the following keywords: KG fusion, multi-source knowledge fusion, KG resolution, KG quality, knowledge fusion, KG reconciliation, KG alignment, KG matching, KG resolution, KG cleaning. The aforementioned keywords have been combined with the keyword “uncertain” and the terms “knowledge”, “data”, and “information” used interchangeably, e.g., “uncertain knowledge fusion”, “uncertain data fusion”, or “uncertain information fusion”.

For the selection of the papers, we proceeded as follows: (1) we look at the title of the paper, if it is relevant and seems to be related to one of the topics we were looking for, then we read the abstract; (2) if the paper presents a method related to uncertain KG construction or a method that is not related to KGs but which could be extrapolated to a KG, we selected it. We provide Figure 2 that depicts the distribution of paper publication years according to four applications of KG refinement: uncertain embedding, knowledge fusion, knowledge alignment, and uncertainty representation. We observe that the representation of uncertainty has been addressed in the literature since 2004, while the vector space representation of uncertainty is a more recent research topic (since 2016). The number of papers on knowledge fusion mentioned in this survey has remained constant over the last few years. This is due to new models based on deep learning that are being explored to tackle these tasks.

<figure><img src="image_4.png" /><figcaption>Fig. 2. Distribution of paper publication years according to uncertain KG embedding (■), knowledge fusion (■), knowledge alignment (■), uncertainty representation (■).</figcaption></figure>

## 3. Knowledge Graphs

Before going into further detail on reconciliation approaches, it is important to define KGs, which are the core of this survey. KGs provide a structural representation of knowledge that is captured by the relations between entities in the graph. The KGs provide a concise and intuitive data representation and abstraction, making them an ideal tool to manage knowledge of organizations in a sustainable way or to support search and querying applications [60], which led several companies to build their own KGs [109].

In this section, we provide a definition of KG, and we describe some well-known KGs including open KGs and Enterprise Knowledge Graphs (EKGs) and how their consistency is maintained in Section 3.1.
### 3.1. What is a Knowledge Graph?

What is the meaning of knowledge? Data are uninterpretable signals (e.g., numbers or characters). Information is data equipped with a meaning. In [122], the authors define knowledge as data and information that enter into a generative process supporting tasks and creating new information. A knowledge graph (KG) is “a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent potentially different relations between these entities” [60].

Formally, KGs are directed and labeled multigraphs ($\mathcal{E}, \mathcal{R}, \mathcal{T}$), where $\mathcal{E}$ is the set of entities, $\mathcal{R}$ is the set of relations, and $\mathcal{T}$ is the set of triples (subject, predicate, object) that are the atomic elements of KGs, also called facts. The subject and object are represented by nodes, the predicate indicates the nature of the relationship holding between the subject and object represented by an edge in the KG [60, 109]. For instance, such a triple could be (Suwon-si, location, Korea) as illustrated in Figure 3. The classes and relationships of entities are defined through a schema or otherwise named an ontology, which can be itself represented as a graph embedded in the KG [30, 36].

In a KG, two boxes coexist, namely Terminology Box (TBox) and Assertion Box (ABox). TBox defines classes and properties and ABox contains instances of classes defined in the TBox. For example, in Figure 3, “Company” and “Country” are concepts defined by the ontology, “Galaxy S23”, “Samsung”, and “Korea” are the instances of these concepts while “2023”, “800€” are literals i.e., attributes that characterize an entity. This data representation in semi-structured form, defined by its ontology, offers a clear and flexible semantic representation whose classes and relations can easily be added and connect a large number of domains [30].

<figure><img src="image_6.png" /><figcaption>Fig. 3. TBox stands for terminology box that contains classes and properties; ABox stands for assertion box that contains instances (e.g., Galaxy S23) and values (e.g., “2023”).</figcaption></figure>

### 3.2. Open Knowledge Graphs

For the last few years, some KG construction projects that link general knowledge about the world have appeared. The best known is probably **Wikidata**<fnref n="3" />, a large, free, and collaborative KG supported by the Wikimedia Foundation. It is a multilingual general-purpose KG that contains more than 100 million elements [138]. The structure of Wikidata is based on property-value pairs, where each property and entity is an element. A typical entity also contains labels, aliases, descriptions, links and mentions to Wikipedia articles. To maintain the quality of data, some constraints such as properties or unique values constraints alert the user in a case of suspicion of the input content

<footnote n="3">³https://www.wikidata.org/wiki/Wikidata:Main_Page</footnote>
### 3.1. What is a Knowledge Graph?

What is the meaning of knowledge? Data are uninterpretable signals (e.g., numbers or characters). Information is data equipped with a meaning. In [122], the authors define knowledge as data and information that enter into a generative process supporting tasks and creating new information. A knowledge graph (KG) is “a graph of data intended to accumulate and convey knowledge of the real world, whose nodes represent entities of interest and whose edges represent potentially different relations between these entities” [60].

Formally, KGs are directed and labeled multigraphs ($\mathcal{E}, \mathcal{R}, \mathcal{T}$), where $\mathcal{E}$ is the set of entities, $\mathcal{R}$ is the set of relations, and $\mathcal{T}$ is the set of triples (subject, predicate, object) that are the atomic elements of KGs, also called facts. The subject and object are represented by nodes, the predicate indicates the nature of the relationship holding between the subject and object represented by an edge in the KG [60, 109]. For instance, such a triple could be (Suwon-si, location, Korea) as illustrated in Figure 3. The classes and relationships of entities are defined through a schema or otherwise named an ontology, which can be itself represented as a graph embedded in the KG [30, 36].

In a KG, two boxes coexist, namely Terminology Box (TBox) and Assertion Box (ABox). TBox defines classes and properties and ABox contains instances of classes defined in the TBox. For example, in Figure 3, “Company” and “Country” are concepts defined by the ontology, “Galaxy S23”, “Samsung”, and “Korea” are the instances of these concepts while “2023”, “800€” are literals i.e., attributes that characterize an entity. This data representation in semi-structured form, defined by its ontology, offers a clear and flexible semantic representation whose classes and relations can easily be added and connect a large number of domains [30].

<figure><img src="image_5.png" /><figcaption>Fig. 3. TBox stands for terminology box that contains classes and properties; ABox stands for assertion box that contains instances (e.g., Galaxy S23) and values (e.g., “2023”).</figcaption></figure>

### 3.2. Open Knowledge Graphs

For the last few years, some KG construction projects that link general knowledge about the world have appeared. The best known is probably **Wikidata**<fnref n="3" />, a large, free, and collaborative KG supported by the Wikimedia Foundation. It is a multilingual general-purpose KG that contains more than 100 million elements [138]. The structure of Wikidata is based on property-value pairs, where each property and entity is an element. A typical entity also contains labels, aliases, descriptions, links and mentions to Wikipedia articles. To maintain the quality of data, some constraints such as properties or unique values constraints alert the user in a case of suspicion of the input content

<footnote n="3">³https://www.wikidata.org/wiki/Wikidata:Main_Page</footnote>
(e.g., constraint violations). Wikidata also keeps the sources and references of provenance of entities to ensure their
traceability, which is one of the requirements for KG quality [145].

NELL is a research project led by Carnegie Mellon University in which an intelligent computer agent that ran continuously between January 2010 and September 2018 according to the official NELL website<fnref n="4" /> and that every day extracted knowledge from texts, tables, and lists from the web to feed a Knowledge Base (KB) [16]. To maintain the consistency of the KB, the knowledge integrator of NELL exploits relationships between predicates by respecting mutual exclusion and type checking information. On top of that, NELL components provide a probability for each candidate and a summary of the source supporting it, hence which can be qualified as a probabilistic KB.

YAGO is an ontology built on statements of Wikipedia that combines high coverage with high quality [128]. The data model of YAGO is based on entities and binary relations extracted from WordNet and Wikipedia. A manual evaluation is performed to verify the quality of data. To do this, facts are randomly selected with their respective Wikipedia pages that are used as Ground Truth (GT).

DBpedia is a multilingual knowledge base built by extracting structured information from Wikipedia (e.g., infoboxes) and makes this information accessible on the Web [6]. Since DBpedia is populated from Wikipedia pages in different languages, data retrieved are sometimes conflicting. To manage these conflicts, DBpedia has a module called *Sieve* which performs a quality assessment by computing some metrics such as the “recency” or the “reputation” of data before applying a fusion step based on these dimensions [13, 99].

Freebase is a graph created in 2007 which provides general human knowledge and which aims to be a public directory of world knowledge. A component included in Freebase called Mass Typer, allows a user to complete data and reconcile it semi-automatically with data already present in Freebase by performing three actions: merge, skip, or add the data. Then acquired by Google and used to support systems like Google Search, Google Maps, and Google KG, nowadays, Freebase is closed, and its knowledge has been transferred to Wikidata [9].

ConceptNet is the KG version of the Open Mind Common Sense project that contains information about words from several languages and their roles in natural language. It was built by collecting knowledge from multiple data sources namely Open Mind Common Sense, Wiktionary, games with a purpose for harvesting common knowledge, Open Multilingual WordNet, JMDict, OpenCyc, and DBPedia. Each node corresponds to a word or a sentence, and the relations between nodes are attached to numerical values that intend to represent the level of uncertainty about the relation [126].

3.3. *Enterprise Knowledge Graphs*

EKGs are major assets for companies since they can support various downstream applications including knowl-
edge/vocabulary sharing and reuse, data integration, information system unification, search, or question answer-
ing [44, 109, 123]. This led companies such as Google, Microsoft, Amazon, Facebook, Orange and IBM, to build
their own KGs [67, 109].

For instance, Microsoft built **Bing KG** to answer any kind of question through Bing search engine. With a size of about two billion entities and 55 billion facts according to [109], it contains general information about the world like people, or places and allows users to take actions like watching a video or buying a song. Alternatively, KGs can also increase understanding of user behavior.

It is the case of the **Facebook KG** that establishes links between the users as well as interests of users, e.g., movies or music tastes. The Facebook KG is the largest social graph with about 50 million entities and 500 million statements in 2019. To handle conflicting information, the Facebook KG removes information if the associated confidence is low, otherwise, conflicting information is integrated with its provenance and the estimated confidence of the information.

Yahoo KG [106] offers different services such as a search engine, a discovery system to relate entities, or for
entity recognition in queries and text. To build their KG, they leverage Wikipedia and Freebase as the backbone of
the KG and use various complementary data sources to maximize the relevance, comprehensiveness, correctness,
freshness, and consistency of knowledge. They mainly validate the data w.r.t. the ontology and through a user
interface that enables entities to be corrected and updated.

<footnote n="4">⁴http://rtw.ml.cmu.edu/rtw/</footnote>
(e.g., constraint violations). Wikidata also keeps the sources and references of provenance of entities to ensure their
traceability, which is one of the requirements for KG quality [145].

NELL is a research project led by Carnegie Mellon University in which an intelligent computer agent that ran continuously between January 2010 and September 2018 according to the official NELL website<fnref n="4" /> and that every day extracted knowledge from texts, tables, and lists from the web to feed a Knowledge Base (KB) [16]. To maintain the consistency of the KB, the knowledge integrator of NELL exploits relationships between predicates by respecting mutual exclusion and type checking information. On top of that, NELL components provide a probability for each candidate and a summary of the source supporting it, hence which can be qualified as a probabilistic KB.

YAGO is an ontology built on statements of Wikipedia that combines high coverage with high quality [128]. The data model of YAGO is based on entities and binary relations extracted from WordNet and Wikipedia. A manual evaluation is performed to verify the quality of data. To do this, facts are randomly selected with their respective Wikipedia pages that are used as Ground Truth (GT).

DBpedia is a multilingual knowledge base built by extracting structured information from Wikipedia (e.g., infoboxes) and makes this information accessible on the Web [6]. Since DBpedia is populated from Wikipedia pages in different languages, data retrieved are sometimes conflicting. To manage these conflicts, DBpedia has a module called *Sieve* which performs a quality assessment by computing some metrics such as the “recency” or the “reputation” of data before applying a fusion step based on these dimensions [13, 99].

Freebase is a graph created in 2007 which provides general human knowledge and which aims to be a public directory of world knowledge. A component included in Freebase called Mass Typer, allows a user to complete data and reconcile it semi-automatically with data already present in Freebase by performing three actions: merge, skip, or add the data. Then acquired by Google and used to support systems like Google Search, Google Maps, and Google KG, nowadays, Freebase is closed, and its knowledge has been transferred to Wikidata [9].

ConceptNet is the KG version of the Open Mind Common Sense project that contains information about words from several languages and their roles in natural language. It was built by collecting knowledge from multiple data sources namely Open Mind Common Sense, Wiktionary, games with a purpose for harvesting common knowledge, Open Multilingual WordNet, JMDict, OpenCyc, and DBPedia. Each node corresponds to a word or a sentence, and the relations between nodes are attached to numerical values that intend to represent the level of uncertainty about the relation [126].

3.3. *Enterprise Knowledge Graphs*

EKGs are major assets for companies since they can support various downstream applications including knowl-
edge/vocabulary sharing and reuse, data integration, information system unification, search, or question answer-
ing [44, 109, 123]. This led companies such as Google, Microsoft, Amazon, Facebook, Orange and IBM, to build
their own KGs [67, 109].

For instance, Microsoft built **Bing KG** to answer any kind of question through Bing search engine. With a size of about two billion entities and 55 billion facts according to [109], it contains general information about the world like people, or places and allows users to take actions like watching a video or buying a song. Alternatively, KGs can also increase understanding of user behavior.

It is the case of the **Facebook KG** that establishes links between the users as well as interests of users, e.g., movies or music tastes. The Facebook KG is the largest social graph with about 50 million entities and 500 million statements in 2019. To handle conflicting information, the Facebook KG removes information if the associated confidence is low, otherwise, conflicting information is integrated with its provenance and the estimated confidence of the information.

**Yahoo KG** [106] offers different services such as a search engine, a discovery system to relate entities, or for entity recognition in queries and text. To build their KG, they leverage Wikipedia and Freebase as the backbone of the KG and use various complementary data sources to maximize the relevance, comprehensiveness, correctness, freshness, and consistency of knowledge. They mainly validate the data *w.r.t.* the ontology and through a user interface that enables entities to be corrected and updated.

<footnote n="4">⁴http://rtw.ml.cmu.edu/rtw/</footnote>
Also, Orange bootstraps its KG from a set of terms of interest from a enterprise repository [67]. These terms of interest are aligned with equivalent Wikidata entities before applying an expansion to retrieve the neighborhood to extend their KG. To ensure the quality of this initial KG, pruning methods based on Euclidean distance in the embedding space, degrees of Wikidata entities, or a method based on analogical inference are used [67, 68].

In [109], the authors mention future challenges including disambiguation, knowledge extraction from unstruc-
tured and heterogeneous sources, and knowledge evolution management in the process of KG construction. We
discuss some of these challenges in the next section.

4. Knowledge Acquisition

The previous section introduced some KGs including open KGs and EKGs. To build such KGs, we could rely on knowledge extraction that is the first step of knowledge integration process. In this section, we present what knowledge extraction is and some well-known automatic approaches that contribute to uncertainty in Section 4.1 that extract knowledge from: texts (Section 4.1.1), Web (Section 4.1.2), and Large Language Models (LLMs) with the recent interest in probing methods (Section 4.1.3). Finally, the definition of KG quality and related metrics are provided in Section 4.2.

4.1. Knowledge Extraction

Methods to populate a KG depend on the knowledge domain and the desired graph coverage. For example, a first method could rely on the knowledge of domain experts and populate the graph manually (e.g., by crowdsourcing such as Wikidata [138] or Freebase [9]). However, this is a time-consuming process, particularly if the graph is intended to be large and it can suffer from quality issues [12, 124]. Furthermore, such open KGs have a large community that enterprises or specific KGs may not have. Therefore, large KGs such as Google, Amazon and Bing KGs rely on automatic construction methods [109]. In the following sections, we present the different tasks involved in knowledge extraction from different types of data sources.

4.1.1. *From texts*

For a long time, the majority of data was represented and exchanged in the form of text [95, 112]. Texts in all their forms (e.g. reports, articles, or any other textual document) are an invaluable source of information, as they are the most widely used data formats in the world (e.g., the scientific research area, where knowledge is communicated via scientific articles [65]). To leverage knowledge from texts as data sources to enrich a KG, we rely on the task called Information Extraction (IE) (or knowledge acquisition). IE transforms unstructured information in text form into structured information, i.e., (*entity*₁, *relation*, *entity*₂) triples [107]. The aim of information extraction is to identify entities, their attributes, and their relationships with other entities in text [148]. In general, this task is separated into several sub-tasks: Entity Recognition (ER) or Named Entity Recognition (NER) and Relation Extraction (RE). Figure 4 depicts the input and output of a text-based knowledge extraction task. NER aims to identify named entities into the text and classify them to general types, while RE extracts semantic relationships that occur at least between two entities [49, 167].

<figure><img src="image_7.png" /><figcaption>Fig. 4. Illustration of knowledge extraction from a single sentence. LOD stands for Linked Open Data.</figcaption></figure>

There are two main IE approaches in the literature [107]: Traditional Information Extraction (Traditional IE) and Open Information Extraction (Open IE). Traditional IE relies on manually defined extraction patterns or patterns
Also, Orange bootstraps its KG from a set of terms of interest from an enterprise repository [67]. These terms of interest are aligned with equivalent Wikidata entities before applying an expansion to retrieve the neighborhood to extend their KG. To ensure the quality of this initial KG, pruning methods based on Euclidean distance in the embedding space, degrees of Wikidata entities, or a method based on analogical inference are used [67, 68].

In [109], the authors mention future challenges including disambiguation, knowledge extraction from unstructured and heterogeneous sources, and knowledge evolution management in the process of KG construction. We discuss some of these challenges in the next section.

# 4. Knowledge Acquisition

The previous section introduced some KGs including open KGs and EKGs. To build such KGs, we could rely on knowledge extraction that is the first step of knowledge integration process. In this section, we present what knowledge extraction is and some well-known automatic approaches that contribute to uncertainty in Section 4.1 that extract knowledge from: texts (Section 4.1.1), Web (Section 4.1.2), and Large Language Models (LLMs) with the recent interest in probing methods (Section 4.1.3). Finally, the definition of KG quality and related metrics are provided in Section 4.2.

## 4.1. Knowledge Extraction

Methods to populate a KG depend on the knowledge domain and the desired graph coverage. For example, a first method could rely on the knowledge of domain experts and populate the graph manually (e.g., by crowdsourcing such as Wikidata [138] or Freebase [9]). However, this is a time-consuming process, particularly if the graph is intended to be large and it can suffer from quality issues [12, 124]. Furthermore, such open KGs have a large community that enterprises or specific KGs may not have. Therefore, large KGs such as Google, Amazon and Bing KGs rely on automatic construction methods [109]. In the following sections, we present the different tasks involved in knowledge extraction from different types of data sources.

### 4.1.1. From texts

For a long time, the majority of data was represented and exchanged in the form of text [95, 112]. Texts in all their forms (e.g. reports, articles, or any other textual document) are an invaluable source of information, as they are the most widely used data formats in the world (e.g., the scientific research area, where knowledge is communicated via scientific articles [65]). To leverage knowledge from texts as data sources to enrich a KG, we rely on the task called Information Extraction (IE) (or knowledge acquisition). IE transforms unstructured information in text form into structured information, i.e., (*entity*₁, *relation*, *entity*₂) triples [107]. The aim of information extraction is to identify entities, their attributes, and their relationships with other entities in text [148]. In general, this task is separated into several sub-tasks: Entity Recognition (ER) or Named Entity Recognition (NER) and Relation Extraction (RE). Figure 4 depicts the input and output of a text-based knowledge extraction task. NER aims to identify named entities into the text and classify them to general types, while RE extracts semantic relationships that occur at least between two entities [49, 167].

<figure><img src="image_6.png" /><figcaption>Fig. 4. Illustration of knowledge extraction from a single sentence. LOD stands for Linked Open Data.</figcaption></figure>

There are two main IE approaches in the literature [107]: Traditional Information Extraction (Traditional IE) and Open Information Extraction (Open IE). Traditional IE relies on manually defined extraction patterns or patterns
learned from manually labeled training examples [107]. However, if the domain of interest evolves, the user must redefine the extraction patterns. Open IE does not rely on predefined patterns and faces three challenges [107, 157]: automation, text heterogeneity and scalability. Automation means that the information extraction system must rely on unsupervised extraction strategies. Heterogeneity stands for the difference of text types across knowledge, e.g., a scholar journal versus a popular science journal. Because the extractions are performed in an unsupervised manner i.e., without any labelling or predefined schema to support the extraction, it implies higher uncertainty in the extracted knowledge. Furthermore, as text types are heterogeneous due to its unstructured form, knowledge extraction patterns are more general and can lead to different levels of specificity of knowledge. Finally, the system must be able to handle large volumes of text for scalability reasons. The most widespread approaches to tackling these issues consist of pipelines composed of methods based on Natural Language Processing (NLP) [148].

One of the earliest examples of a Traditional IE systems is **KnowItAll** [37] that automates the domain-independent extraction of large collections of facts (i.e., triples) from the Web. It is nevertheless supported by an extensible ontology and a minimal set of generic rules to extract entities and relations contained in its ontology. KnowItAll includes four components: an extractor, a search system, an evaluator, and a database. Its extractor instantiates a set of extraction rules for each class and relation based on a generic domain-independent pattern, for example "cities such ..." → "cities such as Paris, Stockholm, ..." deduces that Paris and Stockholm are instances of a "City" class. The search component, which includes 12 search engines such as Google, applies queries based on the extraction rules, i.e., "Cities such as", then retrieves the web pages and applies the extractor. An evaluator leverages the statistics provided by search engines to assess the probability that the extracted relationships are valid. Once the extracted data has passed through these three components, it is stored into a relational database.

Traditional approaches for information extraction rely on an extractor for each target relation based on labeled training examples (e.g., pre-designed extraction patterns). However, these approaches do not address the problem of extraction on large corpora whose relations are not all specified in advance [40] whereas Open IE no longer relies on predefined patterns and allows new information to be explored [107].

For example, **TextRunner** [157] that introduced the concept of Open IE, extracts a set of relational tuples without human input required. TextRunner is described by three components. The first one is a single-pass extractor that labels the text with part-of-speech tags (PoS) (i.e., grammatical tagging) and extracts ($e_1, r, e_2$) triples. The second component is a self-supervised classifier trained to detect the correctness of the extraction. Finally, the last component is a synonym resolver that groups together synonymous entities and relations, since TextRunner has no predefined relations on which extractions are guided.

A slightly more recent approach is **ReVerb** [40]. Using constraints, this method aims to resolve the inconsistent extractions of previous Open IE models due to predicates composed of a verb and a noun. Two types of constraints are introduced on relational sentences: a syntactic constraint and a lexical constraint. Firstly, the syntactic constraint imposes the relational sentence to start either with a verb, a verb followed by a noun, or a verb followed by nouns, adjectives, or adverbs. Regarding the lexical constraint, it focuses on relations that can take many arguments and not on very specific relations. According to the results, these additional constraints allow ReVerb to outperform TextRunner. In addition, ReVerb assigns a confidence score to extractions from a sentence by applying logistic regression classification. To do this, extractions of the form $(x, r, y)$ from a sentence *s* and for 1000 sentences were labeled as valid or invalid, and 19 features such as "$s$ begins with $x$", "$x$ is a proper noun", "$(x, r, y)$ covers all words in *s*" were used as input variables for the logistic regression model. Such confidence scores can be used for downstream knowledge extraction tasks to support their integration into the KG (see Section 5.2).

**OLLIE** [96] expands the syntactic scope of relations phrases to cover much larger number of expressions and allows additional context information such as attribution and clausal modifiers. The authors argue that other models lack context on extracted relations. Hence, compared to previous methods, OLLIE introduces a new component that analyzes the context of an extraction when the extracted relation is not mentioned as factual in the text. This context is attached to each extracted relation and models the validity of the information expressed (e.g., mentions of "*according to*" in a sentence). In [66], the authors present multiple components involved in different IE pipelines in the literature. They propose several combinations of these components and evaluate them in a complete pipeline that includes four steps in the PLUMBER framework: Coreference Resolution, Triples Extraction, Entity Linking and Relation Linking. 40 reusable search components are combined, representing 432 distinct information extraction pipelines. Further information is provided in [66].
learned from manually labeled training examples [107]. However, if the domain of interest evolves, the user must redefine the extraction patterns. Open IE does not rely on predefined patterns and faces three challenges [107, 157]: automation, text heterogeneity and scalability. Automation means that the information extraction system must rely on unsupervised extraction strategies. Heterogeneity stands for the difference of text types across knowledge, e.g., a scholar journal versus a popular science journal. Because the extractions are performed in an unsupervised manner i.e., without any labelling or predefined schema to support the extraction, it implies higher uncertainty in the extracted knowledge. Furthermore, as text types are heterogeneous due to its unstructured form, knowledge extraction patterns are more general and can lead to different levels of specificity of knowledge. Finally, the system must be able to handle large volumes of text for scalability reasons. The most widespread approaches to tackling these issues consist of pipelines composed of methods based on Natural Language Processing (NLP) [148].

One of the earliest examples of a Traditional IE systems is **KnowItAll** [37] that automates the domain-independent extraction of large collections of facts (i.e., triples) from the Web. It is nevertheless supported by an extensible ontology and a minimal set of generic rules to extract entities and relations contained in its ontology. KnowItAll includes four components: an extractor, a search system, an evaluator, and a database. Its extractor instantiates a set of extraction rules for each class and relation based on a generic domain-independent pattern, for example “cities such ...” → “cities such as Paris, Stockholm, ...” deduces that Paris and Stockholm are instances of a “City” class. The search component, which includes 12 search engines such as Google, applies queries based on the extraction rules, i.e., “Cities such as”, then retrieves the web pages and applies the extractor. An evaluator leverages the statistics provided by search engines to assess the probability that the extracted relationships are valid. Once the extracted data has passed through these three components, it is stored into a relational database.

Traditional approaches for information extraction rely on an extractor for each target relation based on labeled training examples (e.g., pre-designed extraction patterns). However, these approaches do not address the problem of extraction on large corpora whose relations are not all specified in advance [40] whereas Open IE no longer relies on predefined patterns and allows new information to be explored [107].

For example, **TextRunner** [157] that introduced the concept of Open IE, extracts a set of relational tuples without human input required. TextRunner is described by three components. The first one is a single-pass extractor that labels the text with part-of-speech tags (PoS) (i.e., grammatical tagging) and extracts ($e_1, r, e_2$) triples. The second component is a self-supervised classifier trained to detect the correctness of the extraction. Finally, the last component is a synonym resolver that groups together synonymous entities and relations, since TextRunner has no predefined relations on which extractions are guided.

A slightly more recent approach is **ReVerb** [40]. Using constraints, this method aims to resolve the inconsistent extractions of previous Open IE models due to predicates composed of a verb and a noun. Two types of constraints are introduced on relational sentences: a syntactic constraint and a lexical constraint. Firstly, the syntactic constraint imposes the relational sentence to start either with a verb, a verb followed by a noun, or a verb followed by nouns, adjectives, or adverbs. Regarding the lexical constraint, it focuses on relations that can take many arguments and not on very specific relations. According to the results, these additional constraints allow ReVerb to outperform TextRunner. In addition, ReVerb assigns a confidence score to extractions from a sentence by applying logistic regression classification. To do this, extractions of the form $(x, r, y)$ from a sentence *s* and for 1000 sentences were labeled as valid or invalid, and 19 features such as “*s* begins with *x*”, “*x* is a proper noun”, “$(x, r, y)$ covers all words in *s*” were used as input variables for the logistic regression model. Such confidence scores can be used for downstream knowledge extraction tasks to support their integration into the KG (see Section 5.2).

**OLLIE** [96] expands the syntactic scope of relations phrases to cover much larger number of expressions and allows additional context information such as attribution and clausal modifiers. The authors argue that other models lack context on extracted relations. Hence, compared to previous methods, OLLIE introduces a new component that analyzes the context of an extraction when the extracted relation is not mentioned as factual in the text. This context is attached to each extracted relation and models the validity of the information expressed (e.g., mentions of “*according to*” in a sentence). In [66], the authors present multiple components involved in different IE pipelines in the literature. They propose several combinations of these components and evaluate them in a complete pipeline that includes four steps in the PLUMBER framework: Coreference Resolution, Triples Extraction, Entity Linking and Relation Linking. 40 reusable search components are combined, representing 432 distinct information extraction pipelines. Further information is provided in [66].
### 4.1.2. From the Web

The Web contains an huge amount of data. It is probably the most widely used tool for exchanging knowledge between people (e.g., in the form of HTML texts). Therefore, it represents an invaluable data source for building KGs. However, the latter suffers from uncertain facts, in part due to the fact that anyone can edit it. In this context, it is necessary to select reliable data sources from the Web and to implement approaches for assessing the reliability of the extracted knowledge. In this section, we present some KGs that have been built from the Web.

NELL [100] has been extracting facts from the Web continuously since January 2010, and aims to improve over time. NELL is a system that takes an initial ontology as input and reads facts from the Web and removes the incorrect ones from a set of labeled data and user feedback on the trustworthiness of the extracted facts. The core of NELL consists of learning thousands of tasks to classify extracted noun phrases into categories, to find the confident relations for each pair of noun phrases, and to identify synonymous noun phrases. Then, the extracted facts are stored in a KB with their provenance and confidence score computed during the relation classification step.

Knowledge Vault [35] is a probabilistic KB that combines extractions from Web content and prior knowledge derived from existing knowledge repositories such as Freebase. They rely on the Local Closed World Assumption, i.e., for a set of existing object values $O(s, p)$ from an existing KG that contains a set of $(s, p, o)$ triples, a candidate triple $(s, p, o)$ is correct if $(s, p, o) \in O(s, p)$. However, if $(s, p, o) \notin O(s, p)$ and $|O(s, p)| > 0$, the triple is incorrect. Hence, this assumption can be difficult to adopt in the construction of an EKG. To merge the extractors (four different fact extraction methods: text documents, HTML trees, HTML tables, and Human Annotated pages) they define a feature vector $f$ for each extracted triple, then apply a binary classifier to compute the probability that the fact is true. They assume that the confidence scores from each extractor are not necessarily on the same scale. Therefore, to cope with this issue, they apply a Platt scaling method that fits a logistic regression model to the confidence scores in order to obtain a probability distribution. Concerning the fusion task, they construct a feature vector $f$ for each extracted triple and apply a binary classifier to compute the probability of the fact to be true given the feature vector. Each predicate is associated to a distinct classifier. Each feature vector contains the square root of the number of sources where the extractor extracted this triple and the mean score of the extractions from this extractor.

Probase [147] does not consider knowledge extracted from the Web to be deterministic, but models it using probabilities. The authors argue that existing KBs and taxonomy construction methods do not have sufficient concept coverage for a machine to understand the text in natural language. Probase includes the uncertainties of the extracted knowledge (specifically vagueness and inconsistencies that are due to the knowledge and to flawed construction methods). It was built from 1.6 billion web pages from an iterative learning algorithm that extracts pairs $(x, y)$ that verify an *isA* relation between *x* and *y*, then a taxonomy construction algorithm organizes these extracted pairs into a hierarchy. In Probase, facts have probabilities that measure their plausibility and typicality. Plausibility is computed from multiple features e.g., the PageRank score, the patterns used to extract *isA* pairs, or the number of sentences where *x* or *y* is present with its respective role (sub or super concept). Typicality is then computed as a function of plausibility and the number of evidences of the fact, i.e., the number of sentences in which the fact is mentioned.

### 4.1.3. Probing

With the arrival of Deep Learning (DL) models and Large Language Models (LLMs), some triple extraction tasks are now successfully carried out by such models. [103] reviews some of them such as Graph-Based Neural Models, CNN-based model, Attention-Based Neural model and others applied to a specific knowledge domain. Also, with significant advances in LLMs and the fact that they are trained on a wide variety of information sources, some researchers have shifted their attention to KG construction by leveraging the knowledge learned by LLMs [112]. For example, a workshop on KB construction from pre-trained language models (KBC-LM<fnref n="5" />) and a challenge on language models for KB construction (LM-KBC<fnref n="6" />) are now proposed at the International Semantic Web Conference (ISWC).

In [49] the authors use the BERT model for NER and RE tasks to build a biomedical KG. In [48], the authors exploit knowledge encoded in LLM parameters (a.k.a. parametric knowledge [112]) to feed a KG by harvesting

<footnote n="5">⁵https://lm-kbc.github.io/workshop2024/</footnote>

<footnote n="6">⁶https://lm-kbc.github.io/challenge2024/</footnote>
### 4.1.2. From the Web

The Web contains an huge amount of data. It is probably the most widely used tool for exchanging knowledge between people (e.g., in the form of HTML texts). Therefore, it represents an invaluable data source for building KGs. However, the latter suffers from uncertain facts, in part due to the fact that anyone can edit it. In this context, it is necessary to select reliable data sources from the Web and to implement approaches for assessing the reliability of the extracted knowledge. In this section, we present some KGs that have been built from the Web.

NELL [100] has been extracting facts from the Web continuously since January 2010, and aims to improve over time. NELL is a system that takes an initial ontology as input and reads facts from the Web and removes the incorrect ones from a set of labeled data and user feedback on the trustworthiness of the extracted facts. The core of NELL consists of learning thousands of tasks to classify extracted noun phrases into categories, to find the confident relations for each pair of noun phrases, and to identify synonymous noun phrases. Then, the extracted facts are stored in a KB with their provenance and confidence score computed during the relation classification step.

Knowledge Vault [35] is a probabilistic KB that combines extractions from Web content and prior knowledge derived from existing knowledge repositories such as Freebase. They rely on the Local Closed World Assumption, i.e., for a set of existing object values $O(s, p)$ from an existing KG that contains a set of $(s, p, o)$ triples, a candidate triple $(s, p, o)$ is correct if $(s, p, o) \in O(s, p)$. However, if $(s, p, o) \notin O(s, p)$ and $|O(s, p)| > 0$, the triple is incorrect. Hence, this assumption can be difficult to adopt in the construction of an EKG. To merge the extractors (four different fact extraction methods: text documents, HTML trees, HTML tables, and Human Annotated pages) they define a feature vector $f$ for each extracted triple, then apply a binary classifier to compute the probability that the fact is true. They assume that the confidence scores from each extractor are not necessarily on the same scale. Therefore, to cope with this issue, they apply a Platt scaling method that fits a logistic regression model to the confidence scores in order to obtain a probability distribution. Concerning the fusion task, they construct a feature vector $f$ for each extracted triple and apply a binary classifier to compute the probability of the fact to be true given the feature vector. Each predicate is associated to a distinct classifier. Each feature vector contains the square root of the number of sources where the extractor extracted this triple and the mean score of the extractions from this extractor.

Probase [147] does not consider knowledge extracted from the Web to be deterministic, but models it using probabilities. The authors argue that existing KBs and taxonomy construction methods do not have sufficient concept coverage for a machine to understand the text in natural language. Probase includes the uncertainties of the extracted knowledge (specifically vagueness and inconsistencies that are due to the knowledge and to flawed construction methods). It was built from 1.6 billion web pages from an iterative learning algorithm that extracts pairs $(x, y)$ that verify an *isA* relation between *x* and *y*, then a taxonomy construction algorithm organizes these extracted pairs into a hierarchy. In Probase, facts have probabilities that measure their plausibility and typicality. Plausibility is computed from multiple features e.g., the PageRank score, the patterns used to extract *isA* pairs, or the number of sentences where *x* or *y* is present with its respective role (sub or super concept). Typicality is then computed as a function of plausibility and the number of evidences of the fact, i.e., the number of sentences in which the fact is mentioned.

### 4.1.3. Probing

With the arrival of Deep Learning (DL) models and Large Language Models (LLMs), some triple extraction tasks are now successfully carried out by such models. [103] reviews some of them such as Graph-Based Neural Models, CNN-based model, Attention-Based Neural model and others applied to a specific knowledge domain. Also, with significant advances in LLMs and the fact that they are trained on a wide variety of information sources, some researchers have shifted their attention to KG construction by leveraging the knowledge learned by LLMs [112]. For example, a workshop on KB construction from pre-trained language models (KBC-LM<fnref n="5" />) and a challenge on language models for KB construction (LM-KBC<fnref n="6" />) are now proposed at the International Semantic Web Conference (ISWC).

In [49] the authors use the BERT model for NER and RE tasks to build a biomedical KG. In [48], the authors exploit knowledge encoded in LLM parameters (a.k.a. parametric knowledge [112]) to feed a KG by harvesting

<footnote n="5">⁵https://lm-kbc.github.io/workshop2024/</footnote>

<footnote n="6">⁶https://lm-kbc.github.io/challenge2024/</footnote>
knowledge for relations of interest. To illustrate their method, they provide an example of knowledge extraction for the “potential_risk” relation. The input contains a prompt such as “The potential risk of A is B” with a few shot of seed entity pairs that validate the relationship, e.g., (eating candy, tooth decay). Then, the entity pairs obtained at the output of the LLM are ranked according to a consistency score computed w.r.t. the compatibility scores between entity pairs. From such point of view, knowledge can be directly extracted from models without visible difficulty by performing aggregation of conflicting information.

However, Pan et al. [112] explore possible interactions and synergies between KGs and LLMs including the construction of KGs from LLMs and raise several issues. LLMs can be used to extract knowledge directly, but this mainly applies to generic, non-specific domains and perform poorly on specific domains. They also lack accuracy in numerical facts such as the birthdate of a person and knowledge of long-tail entities, or have difficulty to memorize them. In addition, LLMs are subject to various biases (e.g., gender bias) that are inherent to training data. Finally, LLMs do not provide any provenance or reliability information of extracted knowledge [112], which can be an obstacle for many knowledge fusion approaches presented in Section 8. In [170], the authors evaluate the ability of LLMs, in particular GPT-3.5, ChatGPT and GPT-4, on KG construction and reasoning (i.e., link prediction and question answering) tasks under different settings, namely zero-shot and one-shot. The authors also point out LLMs fail to outperform state-of-the-art models for KG construction and have limitations to recognize long-tail knowledge.

## 4.2. Quality and metrics

Assessing the quality of the KG constructed is important since it is practically impossible to obtain a perfect KG, especially when this latter is very large and populated by automatic approaches from multiple data sources or by manual approaches where human contributors are not necessarily familiar with KGs and have different levels of expertise. Furthermore, the world is uncertain and knowledge is constantly evolving. To evaluate a KG, we can rely on five quality dimensions [59, 60, 143, 153]: completeness, accuracy, timeliness, availability, and redundancy.

**Completeness** refers to the coverage of knowledge for the specific domain the KG is intended to represent.

**Accuracy** corresponds to the correctness of facts in the KG. In [145], Weikum et al. define some metrics to assess the quality of a KB such as *precision* that captures the accuracy (these terms are sometimes used to describe the same thing), and *recall* that captures the completeness, in the following way:

$$precision(S) = \frac{S \cap GT}{S} \quad \text{and} \quad recall(S) = \frac{S \cap GT}{GT}$$

where *S* is a set of statements from the KB to be evaluated, and *GT* is the ground-truth set for the domain of interest. To deal with uncertain statements that are associated with a confidence score, a threshold is chosen, for which all statements with a score above this threshold are kept. They also provide an evaluation method that involves uniformly taking a sample of statements and representative of the KB and evaluating it, for example manually, where several annotators may be involved and a consensus or large majority must be found for each annotation.

**Timeliness** represents how up-to-date the KG is. The KG can contain temporal facts or facts that evolve and are valid only over a fixed period of time.

**Availability** measures the access to KG data, involving its querying and representation.

**Redundancy** assesses whether different statements express the same fact, which may entail an entity resolution task.

Another aspect of data quality is the preservation and representation of its provenance and certainty in the form of metadata, which can be used for questions of data selection in relation to both source and quality. The metadata can also support knowledge fusion approaches by taking them as prior knowledge, as we describe in Section 8. Other metrics are proposed in the survey [143] for each quality dimension.

# 5. Knowledge Graph refinement

As presented in Section 4, several approaches can be used to extract knowledge from heterogeneous sources (e.g., tables, texts, databases, or human effort) to populate a KG. The advantage of using multiple sources is two-fold:
knowledge for relations of interest. To illustrate their method, they provide an example of knowledge extraction for the “potential_risk” relation. The input contains a prompt such as “The potential risk of A is B” with a few shot of seed entity pairs that validate the relationship, e.g., (eating candy, tooth decay). Then, the entity pairs obtained at the output of the LLM are ranked according to a consistency score computed w.r.t. the compatibility scores between entity pairs. From such point of view, knowledge can be directly extracted from models without visible difficulty by performing aggregation of conflicting information.

However, Pan et al. [112] explore possible interactions and synergies between KGs and LLMs including the construction of KGs from LLMs and raise several issues. LLMs can be used to extract knowledge directly, but this mainly applies to generic, non-specific domains and perform poorly on specific domains. They also lack accuracy in numerical facts such as the birthdate of a person and knowledge of long-tail entities, or have difficulty to memorize them. In addition, LLMs are subject to various biases (e.g., gender bias) that are inherent to training data. Finally, LLMs do not provide any provenance or reliability information of extracted knowledge [112], which can be an obstacle for many knowledge fusion approaches presented in Section 8. In [170], the authors evaluate the ability of LLMs, in particular GPT-3.5, ChatGPT and GPT-4, on KG construction and reasoning (i.e., link prediction and question answering) tasks under different settings, namely zero-shot and one-shot. The authors also point out LLMs fail to outperform state-of-the-art models for KG construction and have limitations to recognize long-tail knowledge.

## 4.2. Quality and metrics

Assessing the quality of the KG constructed is important since it is practically impossible to obtain a perfect KG, especially when this latter is very large and populated by automatic approaches from multiple data sources or by manual approaches where human contributors are not necessarily familiar with KGs and have different levels of expertise. Furthermore, the world is uncertain and knowledge is constantly evolving. To evaluate a KG, we can rely on five quality dimensions [59, 60, 143, 153]: completeness, accuracy, timeliness, availability, and redundancy.

**Completeness** refers to the coverage of knowledge for the specific domain the KG is intended to represent.

**Accuracy** corresponds to the correctness of facts in the KG. In [145], Weikum et al. define some metrics to assess the quality of a KB such as *precision* that captures the accuracy (these terms are sometimes used to describe the same thing), and *recall* that captures the completeness, in the following way:

$$precision(S) = \frac{S \cap GT}{S} \quad \text{and} \quad recall(S) = \frac{S \cap GT}{GT}$$

where *S* is a set of statements from the KB to be evaluated, and *GT* is the ground-truth set for the domain of interest. To deal with uncertain statements that are associated with a confidence score, a threshold is chosen, for which all statements with a score above this threshold are kept. They also provide an evaluation method that involves uniformly taking a sample of statements and representative of the KB and evaluating it, for example manually, where several annotators may be involved and a consensus or large majority must be found for each annotation.

**Timeliness** represents how up-to-date the KG is. The KG can contain temporal facts or facts that evolve and are valid only over a fixed period of time.

**Availability** measures the access to KG data, involving its querying and representation.

**Redundancy** assesses whether different statements express the same fact, which may entail an entity resolution task.

Another aspect of data quality is the preservation and representation of its provenance and certainty in the form of metadata, which can be used for questions of data selection in relation to both source and quality. The metadata can also support knowledge fusion approaches by taking them as prior knowledge, as we describe in Section 8. Other metrics are proposed in the survey [143] for each quality dimension.

# 5. Knowledge Graph refinement

As presented in Section 4, several approaches can be used to extract knowledge from heterogeneous sources (e.g., tables, texts, databases, or human effort) to populate a KG. The advantage of using multiple sources is two-fold:
to ensure knowledge coverage and to identify inconsistencies by leveraging collective wisdom [33]. However, the
world is uncertain and data sources are of varying quality leading to uncertain knowledge, which we need to handle
in the integration process w.r.t. quality dimensions listed in Section 4.2. The causes of uncertainty are presented in
Section 5.1. Then, we describe our theoretical data integration pipeline for dealing with knowledge uncertainty to
enrich a KG in Section 5.2.

5.1. Knowledge Deltas

The uncertainty is everywhere in information and can take the form of knowledge deltas between data sources,
according to [29], we adopt this definition of uncertainty in this survey. We distinguish two types of uncertainty:
epistemic, i.e., knowledge about a piece of information is incomplete or unknown; and ontic, i.e., uncertainty is
inherent in the information [139]. The possible causes of uncertainty are [1, 139]: (i) a lack of knowledge; (ii) a
semantic mismatch or a lack of semantic precision and (iii) a lack of machine precision.

Indeed, when building a KG from heterogeneous sources, some kind of deltas of knowledge between sources
may appear. These deltas of knowledge can occur between two data sources on the same subject, e.g., differences in
granularity and contradictions. It is also possible that a data source contradicts itself, a possible way to detect these
deltas is to compare the data source to itself by “reflecting on data patterns or extrapolation to complete missing
information and/or detect wrong ones” according to [28]. On the other hand, duplicates can also occur if the two
data sources provide exactly the same knowledge, which needs to be managed for reasons of scalability and KG
quality.

We use the same definitions of knowledge deltas as [28], and illustrate them with examples from Wikipedia and
Wikidata, depicted in Figure 5. Suppose that *t* is a statement. Among the possible knowledge deltas, we find six
causes:

- **Invalidity**: *t* is invalid. As illustrated in Figure 5 (a), the Wikipedia text of the figure provides incorrect information: the date of renaming of the Paris region to “île-de-France” is invalid in the Wikipedia page<sup>7</sup>;

- **Vagueness**: *t* provides vague, imprecise information. As depicted in Figure 5 (a), the date mentioned on Wikipedia is more vague than the date provided by Wikidata<fnref n="8" /> for the “located in the administrative territorial entity” property which contains additional information such as the day, month, and year;

- **Fuzziness**: *t* states a fuzzy truth, where the range of values is imprecise itself. As depicted in Figure 5 (b), the Wikipedia article<fnref n="9" /> about 5G claims that the network has higher download speeds peaking at 10 Gbit/s without specifying a lower bound;

- **Timeliness**: a data source can provide the statement *t* which is no longer valid at the current time, unlike another source, which can provide an updated version of *t*. As in Figure 5 (c), on the Wikipedia page<fnref n="10" /> of May 10, 2022, “Twitter” has not yet been renamed to “X”. This information has now been changed, otherwise there would have been an updating issue;

- **Ambiguity**: *t* provides multiple interpretations. As shown in Figure 5 (d), Mercury<fnref n="11" /> can be a planet, an element, or a god in mythology;

- **Incompleteness**: *t* provides incomplete information. As in Figure 5 (e), the track listing of the album “Elevate” by the group Imagine Dragons on Wikidata<fnref n="12" /> contains fewer songs than in Wikipedia<fnref n="13" />.

The creation of knowledge deltas can be involuntary or voluntary. An involuntary delta could be the result of
uncertain knowledge about a domain (e.g., popular science article vs expert article), a typing error, or a data source
not up-to-date. A voluntary delta could simply stem from sabotage by a malicious person (for example, spreading
fake news). Deltas are closely related to the quality dimensions of a KG, since they have a direct impact on them.

<footnote n="7">⁷https://en.wikipedia.org/w/index.php?title=Paris&oldid=1197869134</footnote>

<footnote n="8">⁸https://www.wikidata.org/w/index.php?title=Q90&oldid=2058313448</footnote>

<footnote n="9">⁹https://en.wikipedia.org/wiki/5G</footnote>

<footnote n="10">¹⁰https://en.wikipedia.org/w/index.php?title=Twitter,_Inc.&oldid=1087087372</footnote>

<footnote n="11">¹¹https://en.wikipedia.org/wiki/Mercury</footnote>

<footnote n="12">¹²https://www.wikidata.org/w/index.php?title=Q29868187&oldid=2009666363</footnote>

<footnote n="13">¹³https://en.wikipedia.org/w/index.php?title=Evolve_(Imagine_Dragons album)&oldid=1197244329</footnote>
to ensure knowledge coverage and to identify inconsistencies by leveraging collective wisdom [33]. However, the
world is uncertain and data sources are of varying quality leading to uncertain knowledge, which we need to handle
in the integration process w.r.t. quality dimensions listed in Section 4.2. The causes of uncertainty are presented in
Section 5.1. Then, we describe our theoretical data integration pipeline for dealing with knowledge uncertainty to
enrich a KG in Section 5.2.

5.1. Knowledge Deltas

The uncertainty is everywhere in information and can take the form of knowledge deltas between data sources,
according to [29], we adopt this definition of uncertainty in this survey. We distinguish two types of uncertainty:
epistemic, i.e., knowledge about a piece of information is incomplete or unknown; and ontic, i.e., uncertainty is
inherent in the information [139]. The possible causes of uncertainty are [1, 139]: (i) a lack of knowledge; (ii) a
semantic mismatch or a lack of semantic precision and (iii) a lack of machine precision.

Indeed, when building a KG from heterogeneous sources, some kind of deltas of knowledge between sources
may appear. These deltas of knowledge can occur between two data sources on the same subject, e.g., differences in
granularity and contradictions. It is also possible that a data source contradicts itself, a possible way to detect these
deltas is to compare the data source to itself by “reflecting on data patterns or extrapolation to complete missing
information and/or detect wrong ones” according to [28]. On the other hand, duplicates can also occur if the two
data sources provide exactly the same knowledge, which needs to be managed for reasons of scalability and KG
quality.

We use the same definitions of knowledge deltas as [28], and illustrate them with examples from Wikipedia and
Wikidata, depicted in Figure 5. Suppose that *t* is a statement. Among the possible knowledge deltas, we find six
causes:

- **Invalidity**: *t* is invalid. As illustrated in Figure 5 (a), the Wikipedia text of the figure provides incorrect information: the date of renaming of the Paris region to “île-de-France” is invalid in the Wikipedia page<sup>7</sup>;

- **Vagueness**: *t* provides vague, imprecise information. As depicted in Figure 5 (a), the date mentioned on Wikipedia is more vague than the date provided by Wikidata<fnref n="8" /> for the “located in the administrative territorial entity” property which contains additional information such as the day, month, and year;

- **Fuzziness**: *t* states a fuzzy truth, where the range of values is imprecise itself. As depicted in Figure 5 (b), the Wikipedia article<fnref n="9" /> about 5G claims that the network has higher download speeds peaking at 10 Gbit/s without specifying a lower bound;

- **Timeliness**: a data source can provide the statement *t* which is no longer valid at the current time, unlike another source, which can provide an updated version of *t*. As in Figure 5 (c), on the Wikipedia page<fnref n="10" /> of May 10, 2022, “Twitter” has not yet been renamed to “X”. This information has now been changed, otherwise there would have been an updating issue;

- **Ambiguity**: *t* provides multiple interpretations. As shown in Figure 5 (d), Mercury<fnref n="11" /> can be a planet, an element, or a god in mythology;

- **Incompleteness**: *t* provides incomplete information. As in Figure 5 (e), the track listing of the album “Evolve” by the group Imagine Dragons on Wikidata<fnref n="12" /> contains fewer songs than in Wikipedia<fnref n="13" />.

The creation of knowledge deltas can be involuntary or voluntary. An involuntary delta could be the result of
uncertain knowledge about a domain (e.g., popular science article vs expert article), a typing error, or a data source
not up-to-date. A voluntary delta could simply stem from sabotage by a malicious person (for example, spreading
fake news). Deltas are closely related to the quality dimensions of a KG, since they have a direct impact on them.

<footnote n="7">⁷https://en.wikipedia.org/w/index.php?title=Paris&oldid=1197869134</footnote>

<footnote n="8">⁸https://www.wikidata.org/w/index.php?title=Q90&oldid=2058313448</footnote>

<footnote n="9">⁹https://en.wikipedia.org/wiki/5G</footnote>

<footnote n="10">¹⁰https://en.wikipedia.org/w/index.php?title=Twitter,_Inc.&oldid=1087087372</footnote>

<footnote n="11">¹¹https://en.wikipedia.org/wiki/Mercury</footnote>

<footnote n="12">¹²https://www.wikidata.org/w/index.php?title=Q29868187&oldid=2009666363</footnote>

<footnote n="13">¹³https://en.wikipedia.org/w/index.php?title=Evolve_(Imagine_Dragons album)&oldid=1197244329</footnote>
<figure><img src="image_8.png" /><figcaption>Fig. 5. Illustration of the different possible deltas about some topics between English Wikipedia and Wikidata: (a) invalidity + vagueness, (b) fuzziness, (c) timeliness, (d) ambiguity, and (e) incompleteness.</figcaption></figure>

For example, a delta due to the invalidity of an information from a data source directly affects the accuracy of a KG. We propose to classify these kinds of deltas which lead to conflicts in two classes, namely *Granularity* that stands for a difference between two data sources in the specificity of knowledge and *Contradictory* that stands for an incompatibility of knowledge as depicted in Figure 6. We classify *Fuzziness*, *Incompleteness*, and *Vagueness* deltas in the granularity category. These deltas lead to different levels of specificity between knowledge of two data sources. This knowledge is not necessarily false, but may be in conflict *e.g.*, a city vs. a country to describe the location of an event. On the other hand, *Invalidity*, *Ambiguity*, and *Timeliness* deltas lead to contradictory knowledge, where some parts of the knowledge are necessarily false.

<figure><img src="image_9.png" /><figcaption>Fig. 6. Our proposed classification of knowledge deltas by type of resulting conflict (difference of granularity or contradiction).</figcaption></figure>

In [4, 160], the authors assume that uncertainty is a common feature of the knowledge we handle daily. In this sense, exploiting uncertain data sources by ignoring uncertainty to enrich a KG would impact downstream applications of the graph. The life-cycle for exploiting uncertain data sources requires the measure, the quantification, and the integration of uncertainty in the KG. In such a view, the uncertainty should be considered anywhere in the pipeline of data integration, including its representation within the KG. We present our ideal data integration pipeline that tackle the aforementioned requirements in the following section.

## 5.2. Requirements for an ideal data integration pipeline

All ways of enriching a KG (*e.g.*, crowdsourcing, extraction from texts or tables, etc.) are error-prone methods since a human cannot be an expert on every domain involving mistakes and extraction algorithms rarely achieve perfect accuracy/precision. Errors can occur at several stages in the data integration process that encompasses extraction, alignment, or fusion. Probably one of the most natural ways of capturing and quantifying uncertainty
<figure><img src="image_7.png" /><figcaption>Fig. 5. Illustration of the different possible deltas about some topics between English Wikipedia and Wikidata: (a) invalidity + vagueness, (b) fuzziness, (c) timeliness, (d) ambiguity, and (e) incompleteness.</figcaption></figure>

For example, a delta due to the invalidity of an information from a data source directly affects the accuracy of a KG. We propose to classify these kinds of deltas which lead to conflicts in two classes, namely *Granularity* that stands for a difference between two data sources in the specificity of knowledge and *Contradictory* that stands for an incompatibility of knowledge as depicted in Figure 6. We classify *Fuzziness*, *Incompleteness*, and *Vagueness* deltas in the granularity category. These deltas lead to different levels of specificity between knowledge of two data sources. This knowledge is not necessarily false, but may be in conflict *e.g.*, a city vs. a country to describe the location of an event. On the other hand, *Invalidity*, *Ambiguity*, and *Timeliness* deltas lead to contradictory knowledge, where some parts of the knowledge are necessarily false.

<figure><img src="image_8.png" /><figcaption>Fig. 6. Our proposed classification of knowledge deltas by type of resulting conflict (difference of granularity or contradiction).</figcaption></figure>

In [4, 160], the authors assume that uncertainty is a common feature of the knowledge we handle daily. In this sense, exploiting uncertain data sources by ignoring uncertainty to enrich a KG would impact downstream applications of the graph. The life-cycle for exploiting uncertain data sources requires the measure, the quantification, and the integration of uncertainty in the KG. In such a view, the uncertainty should be considered anywhere in the pipeline of data integration, including its representation within the KG. We present our ideal data integration pipeline that tackle the aforementioned requirements in the following section.

## 5.2. Requirements for an ideal data integration pipeline

All ways of enriching a KG (*e.g.*, crowdsourcing, extraction from texts or tables, etc.) are error-prone methods since a human cannot be an expert on every domain involving mistakes and extraction algorithms rarely achieve perfect accuracy/precision. Errors can occur at several stages in the data integration process that encompasses extraction, alignment, or fusion. Probably one of the most natural ways of capturing and quantifying uncertainty
caused by knowledge deltas or the reliability of knowledge integration components is to use confidence scores. As mentioned in Section 4, several extraction approaches provide confidence scores about the triples they extract. For example, each triple outputted by ReVerb [40] is associated to a confidence score obtained from a logistic regression. Another work [86] focuses on estimating a confidence score for the slot filling task, which consists in filling predefined attributes for entities in a KB population case. This confidence score is intended to support the aggregation of values from different slot filling systems. The authors have shown that confidence estimation improves performance of the task and that the correctness of values and estimated confidence are strongly correlated. In [146], the authors estimate confidence scores for an entity alignment task that represents the marginal probability that a set of mentions all refer to a same entity. Therefore, there is a need to consider these confidence scores and represent them as triple metadata along with their provenance information.

<figure><img src="image_10.png" /><figcaption>Fig. 7. Process of integrating uncertain knowledge extracted from heterogeneous sources, ideally linked to different confidence scores. Three steps form knowledge integration: (1) alignment, (2) fusion, and (3) consistency checking.</figcaption></figure>

From this perspective, we propose an ideal pipeline of data integration from heterogeneous sources depicted in Figure 7. In the literature, knowledge integration after extraction is often described in two modules [8]: *Knowledge Alignment* and *Knowledge Fusion*. In this pipeline, we propose a third module called *Consistency Checking*, which actually takes place after the data integration and identifies and repairs inconsistencies in the KG, improving future knowledge enrichment. The inputs of the pipeline are multiple heterogeneous sources whose final purpose is to feed the KG. From these data sources $s_j$, facts $fact_{i,s_j}$ are extracted with different confidence scores such as a confidence score in the fact by the extraction algorithm $conf_{extraction}(fact_{i,s_j})$, a confidence score in the fact by the source $conf_{source}(fact_{i,s_j})$ and a confidence score in the source $conf(s_j)$. In addition to these multiple data sources, an expert can also populate the KG with a confidence score $conf(s_e)$. Before providing these facts to the KG, several tasks are required due to potential knowledge deltas. The first task *Knowledge Alignment* is the identification of duplicates, differences of granularity and contradictions among extracted facts and the KG. The next step *Knowledge Fusion* defines a policy to remove conflicting facts and keeps information as consistent, specific, and complete as possible. Then, knowledge in the KG is updated with their confidence scores and their provenance information since a user might want to query the KG about confidence of triples w.r.t. quality dimensions. A last step verifies the consistency within the KG. Since this step is performed after the enrichment of the KG and that this survey is focused on the uncertainty management in the construction of KGs, we do not provide further details on it in the following sections.

The aim of this pipeline is to take into account all the confidence scores in the knowledge alignment and fusion modules. This is not the case in existing work, where only the confidence scores in the data sources are leveraged
caused by knowledge deltas or the reliability of knowledge integration components is to use confidence scores. As mentioned in Section 4, several extraction approaches provide confidence scores about the triples they extract. For example, each triple outputted by ReVerb [40] is associated to a confidence score obtained from a logistic regression. Another work [86] focuses on estimating a confidence score for the slot filling task, which consists in filling predefined attributes for entities in a KB population case. This confidence score is intended to support the aggregation of values from different slot filling systems. The authors have shown that confidence estimation improves performance of the task and that the correctness of values and estimated confidence are strongly correlated. In [146], the authors estimate confidence scores for an entity alignment task that represents the marginal probability that a set of mentions all refer to a same entity. Therefore, there is a need to consider these confidence scores and represent them as triple metadata along with their provenance information.

<figure><img src="image_9.png" /><figcaption>Fig. 7. Process of integrating uncertain knowledge extracted from heterogeneous sources, ideally linked to different confidence scores. Three steps form knowledge integration: (1) alignment, (2) fusion, and (3) consistency checking.</figcaption></figure>

From this perspective, we propose an ideal pipeline of data integration from heterogeneous sources depicted in Figure 7. In the literature, knowledge integration after extraction is often described in two modules [8]: *Knowledge Alignment* and *Knowledge Fusion*. In this pipeline, we propose a third module called *Consistency Checking*, which actually takes place after the data integration and identifies and repairs inconsistencies in the KG, improving future knowledge enrichment. The inputs of the pipeline are multiple heterogeneous sources whose final purpose is to feed the KG. From these data sources $s_j$, facts $fact_{i,s_j}$ are extracted with different confidence scores such as a confidence score in the fact by the extraction algorithm $conf_{extraction}(fact_{i,s_j})$, a confidence score in the fact by the source $conf_{source}(fact_{i,s_j})$ and a confidence score in the source $conf(s_j)$. In addition to these multiple data sources, an expert can also populate the KG with a confidence score $conf(s_e)$. Before providing these facts to the KG, several tasks are required due to potential knowledge deltas. The first task *Knowledge Alignment* is the identification of duplicates, differences of granularity and contradictions among extracted facts and the KG. The next step *Knowledge Fusion* defines a policy to remove conflicting facts and keeps information as consistent, specific, and complete as possible. Then, knowledge in the KG is updated with their confidence scores and their provenance information since a user might want to query the KG about confidence of triples w.r.t. quality dimensions. A last step verifies the consistency within the KG. Since this step is performed after the enrichment of the KG and that this survey is focused on the uncertainty management in the construction of KGs, we do not provide further details on it in the following sections.

The aim of this pipeline is to take into account all the confidence scores in the knowledge alignment and fusion modules. This is not the case in existing work, where only the confidence scores in the data sources are leveraged
by the fusion module. However, methods for KG completion purpose (i.e., predicting new relations using only the KG itself) taking into account uncertainty in an embedding space has recently been studied.

We describe and formalize the ideal data integration policy with an example. As input, we provide a set of sources $S$ that contain a set of facts $\mathcal{F}$ and we obtain a $KG$ as output. For each fact belonging to the source $f_S$, this fact is aligned with the $KG$. If the fact $f_S$ conflicts with a fact already present in the KG $f_{KG}$, we proceed as follows:

(1) If $f_S$ is more specific than $f_{KG}$ then we replace $f_{KG}$ by $f_S$ and we increase the confidence given to the source $S$. For example, “Joe Biden is president of a North America (NA) country” “Joe Biden is President of the USA” $\rightarrow$ We keep the most specific fact, i.e., “Joe Biden is president of the USA”. But it would be interesting if we leverage the information “NA” by deviating from it $\rightarrow$ “USA is a country in NA” to complete the graph;

(2) Otherwise, if $f_S$ is the duplicate of $f_{KG}$, we only add the provenance of $f_S$ and we increase the trust placed in the source $S$;

(3) Otherwise, if $f_S$ is contradictory to $f_{KG}$, we resolve the conflict by finding the true value and we decrease the source that provides an erroneous fact and increase the source that provides a correct one;

(4) Else, if the fact does not conflict with a fact of $KG$, the fact is added to the KG with the associated confidence scores and provenance.

A formalization of the policy is provided in Algorithm 1.

As mentioned in step (4) in the process above, we need to keep a history of knowledge provenance since the provenance information is necessary for KG quality, but could also be used for future conflict resolution or KG updating [64]. For this purpose, there is a normative ontology called PROV-O [79] that includes provenance information through three components: a set of classes, properties, and restrictions. It can be used in RDF-based KGs. The three main classes of the PROV-O ontology are *prov:Entity*, *prov:Activity*, and *prov:Agent* (*prov:Entity* is something that can be changed by an activity, *prov:Activity* is something that acts upon or with entities, and *prov:Agent* can be a human who performs an activity).

In the following sections, we detail three steps of the pipeline namely: Knowledge alignment, Knowledge Fusion, and Uncertainty Representation within the KG. We propose a section that describes uncertain KG embedding methods for KG completion and confidence prediction tasks (Section 6) before presenting the aforementioned steps. Knowledge alignment is discussed in Section 7. In Section 8, we summarize knowledge fusion methods. Finally, we explore the different mechanisms available for representing triple uncertainty in a KG in Section 9.

# 6. Uncertain Knowledge Graph Embedding

Embedding methods enable KG representation in a $n$-dimensional vectorial space, i.e., its entities and relations are $n$-vectors. These embeddings attempt to preserve the structural properties of the graph and thus make it easy to manipulate the graph for machine learning applications such as link prediction, completion, or node classification [69]. A wide range of embedding models have emerged such as TransE [10], DistMult [154], ComplEx [137], RotatE [133], neural networks applied to graphs such as RGCN [121], or GCN [76]. In addition, embeddings are also increasingly used in the construction of KGs (for example, for knowledge alignment [42], or other tasks for KG refinement [61]). Most embedding approaches do not include the uncertainty of knowledge in their models. However, when constructing KBs, the knowledge is often uncertain or noisy and not taking into account uncertainty during the representation learning can imply a bias in its representation and impact further applications. Given the importance of embedding methods in both KG applications and construction, we consider that it is useful to gather such methods that include uncertainty expressed in terms of a confidence score in their modeling. This section describes some of these models and the datasets used to evaluate them.

## 6.1. Uncertain KG embedding models

In this paper, we formalize uncertain KGs as follows. An *uncertain KG* (UKG) is represented as a set of weighted triples $\mathcal{G} = (s, p, o, s_t)$, where $(s, p, o)$ is a triple representing a fact and $s_t \in [0, 1]$ is a confidence score for this fact to be true. The uncertainty linked to the triples in the KG relies on the plausibility of the triples, but most
by the fusion module. However, methods for KG completion purpose (i.e., predicting new relations using only the KG itself) taking into account uncertainty in an embedding space has recently been studied.

We describe and formalize the ideal data integration policy with an example. As input, we provide a set of sources $S$ that contain a set of facts $\mathcal{F}$ and we obtain a $KG$ as output. For each fact belonging to the source $f_S$, this fact is aligned with the $KG$. If the fact $f_S$ conflicts with a fact already present in the KG $f_{KG}$, we proceed as follows:

(1) If $f_S$ is more specific than $f_{KG}$ then we replace $f_{KG}$ by $f_S$ and we increase the confidence given to the source $S$. For example, “Joe Biden is president of a North America (NA) country” “Joe Biden is President of the USA” $\rightarrow$ We keep the most specific fact, i.e., “Joe Biden is president of the USA”. But it would be interesting if we leverage the information “NA” by deviating from it $\rightarrow$ “USA is a country in NA” to complete the graph;

(2) Otherwise, if $f_S$ is the duplicate of $f_{KG}$, we only add the provenance of $f_S$ and we increase the trust placed in the source $S$;

(3) Otherwise, if $f_S$ is contradictory to $f_{KG}$, we resolve the conflict by finding the true value and we decrease the source that provides an erroneous fact and increase the source that provides a correct one;

(4) Else, if the fact does not conflict with a fact of $KG$, the fact is added to the KG with the associated confidence scores and provenance.

A formalization of the policy is provided in Algorithm 1.

As mentioned in step (4) in the process above, we need to keep a history of knowledge provenance since the provenance information is necessary for KG quality, but could also be used for future conflict resolution or KG updating [64]. For this purpose, there is a normative ontology called PROV-O [79] that includes provenance information through three components: a set of classes, properties, and restrictions. It can be used in RDF-based KGs. The three main classes of the PROV-O ontology are *prov:Entity*, *prov:Activity*, and *prov:Agent* (*prov:Entity* is something that can be changed by an activity, *prov:Activity* is something that acts upon or with entities, and *prov:Agent* can be a human who performs an activity).

In the following sections, we detail three steps of the pipeline namely: Knowledge alignment, Knowledge Fusion, and Uncertainty Representation within the KG. We propose a section that describes uncertain KG embedding methods for KG completion and confidence prediction tasks (Section 6) before presenting the aforementioned steps. Knowledge alignment is discussed in Section 7. In Section 8, we summarize knowledge fusion methods. Finally, we explore the different mechanisms available for representing triple uncertainty in a KG in Section 9.

# 6. Uncertain Knowledge Graph Embedding

Embedding methods enable KG representation in a $n$-dimensional vectorial space, i.e., its entities and relations are $n$-vectors. These embeddings attempt to preserve the structural properties of the graph and thus make it easy to manipulate the graph for machine learning applications such as link prediction, completion, or node classification [69]. A wide range of embedding models have emerged such as TransE [10], DistMult [154], ComplEx [137], RotatE [133], neural networks applied to graphs such as RGCN [121], or GCN [76]. In addition, embeddings are also increasingly used in the construction of KGs (for example, for knowledge alignment [42], or other tasks for KG refinement [61]). Most embedding approaches do not include the uncertainty of knowledge in their models. However, when constructing KBs, the knowledge is often uncertain or noisy and not taking into account uncertainty during the representation learning can imply a bias in its representation and impact further applications. Given the importance of embedding methods in both KG applications and construction, we consider that it is useful to gather such methods that include uncertainty expressed in terms of a confidence score in their modeling. This section describes some of these models and the datasets used to evaluate them.

## 6.1. Uncertain KG embedding models

In this paper, we formalize uncertain KGs as follows. An *uncertain KG* (UKG) is represented as a set of weighted triples $\mathcal{G} = (s, p, o, s_t)$, where $(s, p, o)$ is a triple representing a fact and $s_t \in [0, 1]$ is a confidence score for this fact to be true. The uncertainty linked to the triples in the KG relies on the plausibility of the triples, but most
**Algorithm 1** Data integration policy

**Input:** A set of facts $\mathcal{F}$ from a source $\mathcal{S}$ with confidence $conf(\mathcal{S})$ where each fact $fact_S$ is associated to a confidence by the algorithm of extraction $conf_{extract}(fact_S)$ and a confidence by the source $conf_{source}(fact_S)$, $conf(fact_S) = (conf(\mathcal{S}), conf_{extract}(fact_S), conf_{source}(fact_S))$, and a $KG$

**Output:** $KG$ updated with consistent facts of $\mathcal{S}$

for $f_S \in \mathcal{F}$ do
    $f_{KG} \leftarrow$ align($KG$, $f_S$, $conf(fact_S)$)
    if $f_{KG} \neq \emptyset$ then
        if moreSpecific($f_S$, $f_{KG}$) then
            replace($f_{KG}$, $f_S$)
            increase(conf($\mathcal{S}$))
        else if similar($f_S$, $f_{KG}$) then
            addSource($KG$, $f_{KG}$, $\mathcal{S}$)
            increase(conf($\mathcal{S}$))
        else if contradictory($f_S$, $f_{KG}$) then
            decrease(conf($\mathcal{S}$))
    end if
    else
        $KG \leftarrow KG \cup \{f_S\}$
    end if
end for

KG embedding (KGE) methods do not consider this information in their modeling, making the assumption that all triples are deterministic. Such an assumption does not reflect the reality where many triples are uncertain due to the reasons described in Section 5. Table 1 summarizes the UKG embedding approaches with their associated tasks, scoring function, the year of publication, and the datasets on which the experiments were conducted. We can notice that uncertain graph embeddings have only been recently studied.

UKGE [20] improves traditional KGE models by using the Probabilistic Soft Logic (PSL) framework to infer confidence scores for unseen relational triples. Thus, UKGE encodes the KG according to confidence scores for observed and unseen triples. They map the scoring function results into confidence scores using two different mapping functions, namely a logistic function or the bounded rectifier function. For the relation fact classification, global ranking, and confidence prediction tasks, UKGE outperforms the deterministic KG embedding models such as TransE, DistMult, ComplEx and the URGE model on CN15k, NL27k and PPI5k datasets.

SUKE [141] argues that UKGE does not make full use of the structure information of the fact. Therefore, to improve this, SUKE has two components: an evaluator and a confidence generator. The evaluator defines a structure score and an uncertainty score for each fact to capture the rationality of triples. The generator outputs confidence scores for triples from the uncertainty score computed by the evaluator. The plausibility of facts are computed with DistMult [154] scoring function, then it applies a different mapping function with two parameters for the structural score and the uncertain score before merging them. The confidence generator only uses the uncertainty score to approximate the true confidence value of triples.

BEURRE [21] models entities as probabilistic boxes and relations between two entities as an affine transformation. The confidence score of the relation between two entities is represented as the volume of the intersection of their boxes. Constraints such as transitivity and composition are inserted into the modeling of embeddings to preserve these properties on relations in the embedding space. These constraints act as a loss regularization in the global loss function. Then, embeddings are trained by optimizing a loss function for a regression task and a regularization loss to apply transitivity and composition constraints.

GTransE [74] embeds uncertainty through a translational model by expanding the well known TransE [10]. The uncertainty is included at the level of the loss function on a hyperparameter of the margin loss function when training
**Algorithm 1** Data integration policy

**Input:** A set of facts $\mathcal{F}$ from a source $\mathcal{S}$ with confidence $conf(\mathcal{S})$ where each fact $fact_S$ is associated to a confidence by the algorithm of extraction $conf_{extract}(fact_S)$ and a confidence by the source $conf_{source}(fact_S)$, $conf(fact_S) = (conf(\mathcal{S}), conf_{extract}(fact_S), conf_{source}(fact_S))$, and a $KG$

**Output:** $KG$ updated with consistent facts of $\mathcal{S}$

for $f_S \in \mathcal{F}$ do
    $f_{KG} \leftarrow$ align($KG$, $f_S$, $conf(fact_S)$)
    if $f_{KG} \neq \emptyset$ then
        if moreSpecific($f_S$, $f_{KG}$) then
            replace($f_{KG}$, $f_S$)
            increase(conf($\mathcal{S}$))
        else if similar($f_S$, $f_{KG}$) then
            addSource($KG$, $f_{KG}$, $\mathcal{S}$)
            increase(conf($\mathcal{S}$))
        else if contradictory($f_S$, $f_{KG}$) then
            decrease(conf($\mathcal{S}$))
    end if
    else
        $KG \leftarrow KG \cup \{f_S\}$
    end if
end for

KG embedding (KGE) methods do not consider this information in their modeling, making the assumption that all triples are deterministic. Such an assumption does not reflect the reality where many triples are uncertain due to the reasons described in Section 5. Table 1 summarizes the UKG embedding approaches with their associated tasks, scoring function, the year of publication, and the datasets on which the experiments were conducted. We can notice that uncertain graph embeddings have only been recently studied.

UKGE [20] improves traditional KGE models by using the Probabilistic Soft Logic (PSL) framework to infer confidence scores for unseen relational triples. Thus, UKGE encodes the KG according to confidence scores for observed and unseen triples. They map the scoring function results into confidence scores using two different mapping functions, namely a logistic function or the bounded rectifier function. For the relation fact classification, global ranking, and confidence prediction tasks, UKGE outperforms the deterministic KG embedding models such as TransE, DistMult, ComplEx and the URGE model on CN15k, NL27k and PPI5k datasets.

SUKE [141] argues that UKGE does not make full use of the structure information of the fact. Therefore, to improve this, SUKE has two components: an evaluator and a confidence generator. The evaluator defines a structure score and an uncertainty score for each fact to capture the rationality of triples. The generator outputs confidence scores for triples from the uncertainty score computed by the evaluator. The plausibility of facts are computed with DistMult [154] scoring function, then it applies a different mapping function with two parameters for the structural score and the uncertain score before merging them. The confidence generator only uses the uncertainty score to approximate the true confidence value of triples.

BEURRE [21] models entities as probabilistic boxes and relations between two entities as an affine transformation. The confidence score of the relation between two entities is represented as the volume of the intersection of their boxes. Constraints such as transitivity and composition are inserted into the modeling of embeddings to preserve these properties on relations in the embedding space. These constraints act as a loss regularization in the global loss function. Then, embeddings are trained by optimizing a loss function for a regression task and a regularization loss to apply transitivity and composition constraints.

GTransE [74] embeds uncertainty through a translational model by expanding the well known TransE [10]. The uncertainty is included at the level of the loss function on a hyperparameter of the margin loss function when training
the embeddings:

$$
\begin{align*}
\mathcal{L} &= \sum_{(h,r,t,s) \in Q} \sum_{(h',r,t',s') \in Q'} [f(h,r,t) - f(h',r,t') + s^{\alpha}M]_{+} \\
&= \sum_{(h,r,t,s) \in Q} \sum_{(h',r,t',s') \in Q'} \max 0, f(h,r,t) - f(h',r,t') + s^{\alpha}M
\end{align*}
$$

where $(h, r, t, s)=(\text{head}, \text{relation}, \text{tail}, \text{confidence score})$, $M$ a margin parameter and $[x]_+$ is the positive part of $x$. The scoring function $f$ corresponds to L1 or L2-norm. Thus, with this loss function if a triple $(h, r, t)$ has a high confidence score it will tend to respect $h + r = t$ otherwise the entity $t$ will tend to move away from $h + r$. Before GTransE, the same authors introduced **CTransE** [73] that is closely the same model but without the hyperparameter $\alpha$ in power of the confidence score.

**IIKE** [41] models the confidence in the embedding space through a probabilistic model. The authors propose an embedding model that takes uncertainty into account by minimizing a loss function to fit the output confidence of triples acquisition (e.g., NELL, or crowdsourcing) to the scoring function of the triples given by a probability function. The plausibility of a triple is modeled as a joint probability of the head of the entity, the relation and the tail $Pr(h, r, t)$ depending on $Pr(h|r, t)$, $Pr(r|h, t)$, and $Pr(t|h, r)$. For the loss function, the authors minimize the difference between the logarithm of the triple probabilities and the logarithm of the confidence of the knowledge extraction. Then apply stochastic gradient descent to refine embeddings at each iteration.

**PASSLEAF** [22] decomposes the model in two parts: a confidence score prediction framework that adapts the score function among existing ones, e.g., ComplEx [137] or RotatE [133] and a semi-supervised learning framework.

For the UKG completion task, each relation must have enough training examples to perform correctly. **GMUC** [161] addresses the few-shot UKG completion task for long-tail relations. GMUC learns a Gaussian similarity metric that allows missing facts and their confidence scores to be predicted from a few training examples. The model encodes a support set containing a few facts with their confidence scores and a query into multidimensional Gaussian distributions. The query consists of pairs (*head*, *relation*) where tail and score must be predicted. Then a Gaussian matching function is used to generate a similarity distribution between the query and the support set. GMUC outperforms UKGE model on link prediction and confidence prediction on NL27K and three NL27K-derived datasets with added noise.

**UOKGE** [11] learns embeddings of uncertain ontology-aware KGs according to confidence scores. It encodes an instance $e_i \in E$ as a point represented by a n-dimensional vector, a class $c_i \in C$ as a sphere $s_i(c_i, \rho_i)$ where $c_i$ is the center of the sphere and $p_i$ is the radius, and a property $p_i \in P$ as a sphere $s_i((p_i^d, p_i^r), \varrho_i)$ where $(p_i^d, p_i^r)$ is the center of the sphere with $p_i^d$ representing the domain, $p_i^r$ representing the range, and $\varrho_i$ is the radius. Then, it introduces the following mapping function that allows changing the scale of values between 0 and 1 to represent the uncertainty. Six distinct gap functions for six types of relation are defined to encode uncertainty for: type, domain, range, subclass, sup-property and remaining properties. Then, it minimizes the mean squared error between the confidence score and those gap functions.

**FocusE** [111] improves KG embeddings with numerical values on edges by taking action between the scoring function of traditional models (e.g., TransE, ComplEx, or DistMult) and the loss function. They introduce numerical values on edges in a way that maximizes the margin between the scores associated with true triples and their corruptions. Given the score function of an embedding model $f(t)$ with $t$ a triple, they use a nonlinear softplus $\sigma$ in order that the score provided by $f(t)$ is greater or equal to zero:

$$g(t) = \sigma(f(t)) = \ln(1 + e^{f(t)}) \ge 0$$

Then, the numerical value associated to an edge is expressed through $\alpha$ in the following way:

$$\alpha = \begin{cases} \beta + (1 - w)(1 - \beta) & \text{if } t \\ \beta + w(1 - \beta) & \text{if } t^{-} \end{cases}$$
the embeddings:

$$
\begin{align*}
\mathcal{L} &= \sum_{(h,r,t,s) \in Q} \sum_{(h',r,t',s') \in Q'} [f(h,r,t) - f(h',r,t') + s^{\alpha}M]_{+} \\
&= \sum_{(h,r,t,s) \in Q} \sum_{(h',r,t',s') \in Q'} \max 0, f(h,r,t) - f(h',r,t') + s^{\alpha}M
\end{align*}
$$

where $(h, r, t, s)=(\text{head}, \text{relation}, \text{tail}, \text{confidence score})$, $M$ a margin parameter and $[x]_+$ is the positive part of $x$. The scoring function $f$ corresponds to L1 or L2-norm. Thus, with this loss function if a triple $(h, r, t)$ has a high confidence score it will tend to respect $h + r = t$ otherwise the entity $t$ will tend to move away from $h + r$. Before GTransE, the same authors introduced **CTransE** [73] that is closely the same model but without the hyperparameter $\alpha$ in power of the confidence score.

**IIKE** [41] models the confidence in the embedding space through a probabilistic model. The authors propose an embedding model that takes uncertainty into account by minimizing a loss function to fit the output confidence of triples acquisition (e.g., NELL, or crowdsourcing) to the scoring function of the triples given by a probability function. The plausibility of a triple is modeled as a joint probability of the head of the entity, the relation and the tail $Pr(h, r, t)$ depending on $Pr(h|r, t)$, $Pr(r|h, t)$, and $Pr(t|h, r)$. For the loss function, the authors minimize the difference between the logarithm of the triple probabilities and the logarithm of the confidence of the knowledge extraction. Then apply stochastic gradient descent to refine embeddings at each iteration.

**PASSLEAF** [22] decomposes the model in two parts: a confidence score prediction framework that adapts the score function among existing ones, e.g., ComplEx [137] or RotatE [133] and a semi-supervised learning framework.

For the UKG completion task, each relation must have enough training examples to perform correctly. **GMUC** [161] addresses the few-shot UKG completion task for long-tail relations. GMUC learns a Gaussian similarity metric that allows missing facts and their confidence scores to be predicted from a few training examples. The model encodes a support set containing a few facts with their confidence scores and a query into multidimensional Gaussian distributions. The query consists of pairs (*head*, *relation*) where tail and score must be predicted. Then a Gaussian matching function is used to generate a similarity distribution between the query and the support set. GMUC outperforms UKGE model on link prediction and confidence prediction on NL27K and three NL27K-derived datasets with added noise.

**UOKGE** [11] learns embeddings of uncertain ontology-aware KGs according to confidence scores. It encodes an instance $e_i \in E$ as a point represented by a n-dimensional vector, a class $c_i \in C$ as a sphere $s_i(c_i, \rho_i)$ where $c_i$ is the center of the sphere and $p_i$ is the radius, and a property $p_i \in P$ as a sphere $s_i((p_i^d, p_i^r), \varrho_i)$ where $(p_i^d, p_i^r)$ is the center of the sphere with $p_i^d$ representing the domain, $p_i^r$ representing the range, and $\varrho_i$ is the radius. Then, it introduces the following mapping function that allows changing the scale of values between 0 and 1 to represent the uncertainty. Six distinct gap functions for six types of relation are defined to encode uncertainty for: type, domain, range, subclass, sup-property and remaining properties. Then, it minimizes the mean squared error between the confidence score and those gap functions.

**FocusE** [111] improves KG embeddings with numerical values on edges by taking action between the scoring function of traditional models (e.g., TransE, ComplEx, or DistMult) and the loss function. They introduce numerical values on edges in a way that maximizes the margin between the scores associated with true triples and their corruptions. Given the score function of an embedding model $f(t)$ with $t$ a triple, they use a nonlinear softplus $\sigma$ in order that the score provided by $f(t)$ is greater or equal to zero:

$$g(t) = \sigma(f(t)) = \ln(1 + e^{f(t)}) \ge 0$$

Then, the numerical value associated to an edge is expressed through $\alpha$ in the following way:

$$\alpha = \begin{cases} \beta + (1 - w)(1 - \beta) & \text{if } t \\ \beta + w(1 - \beta) & \text{if } t^{-} \end{cases}$$
where $\beta$ is a hyperparameter acting on the importance of the topological structure of the graph and $w$ is the numerical value on the edge. Then, the final function of FocusE is the following: $h(t) = \alpha g(t)$.

**ConFE [165]** encodes the tuples $(e, \tau)$ with $e$ an entity and $\tau$ an entity type by taking into account the uncertainty in each tuple. They consider entities and entity types as two different things in a KG and learn embeddings of entities and entity types in two distinct spaces with an asymmetric matrix to model their interactions. The scoring function is defined as $G(e, \tau) = e^{\top} M \tau$ where $M$ is the asymmetric matrix. And uncertainty is included within the loss function:

$$ \mathcal{L} = \sum_{(e, \tau) \in \mathcal{H}} \sum_{(e', \tau') \in \mathcal{H}'} \max(0, \gamma - G(e, \tau) + G(e', \tau')).C(e, \tau) $$

where $\mathcal{H}$ is the set of entities and their type and $\mathcal{H}'$ the set of corrupted tuples.

**CKRL [151]** introduces multiple levels of confidence, namely a local triple confidence, a global path confidence, a prior path confidence, and an adaptive path confidence. These confidence scores are integrated into the energy function following designed:

$$ E(T) = \sum_{(h,r,t) \in T} E(h, r, t) \cdot C(h, r, t) $$

where $E(h, r, t) = \|h + r - t\|$ and $C(h, r, t)$ the triple confidence score aggregating all levels of confidence.

**WaExt [77]** embeds triples of a KG by incorporating the weight $w$ associated to an edge $(h, r, t)$ in the scoring function in the following manner:

$$ f_w(h, r, t, w) = g(w) \cdot f(h, r, t) $$

and then minimizes a margin ranking loss function.

**Wang et al. [142]** model each entity and each relation as a multidimensional Gaussian distribution $N(\mu, \Sigma)$ where $\mu$ is a mean vector representing its position and $\Sigma$ is a diagonal covariance matrix representing its uncertainty.

**MUKGE** [91] aims to improve the generation of unseen facts for KGE training. The authors argue that PSL cannot take advantage of global multi-path information involving information loss to estimate the confidence of unseen facts. Indeed, PSL only considers information from simple logical rules with a path length of two as used in UKGE [20] (e.g., (college, synonym, university) ∧ (university, synonym, institute) → (college, synonym, institute)) and other paths in the graph between the subject and object of the inferred relation are not taken into account. To solve this issue, MUKGE introduces an algorithm called Uncertain ResourceRank to infer confidence scores for unseen triples based on the relevance of the entity pairs (subject, object). The relevance of an entity pair is computed w.r.t. the directed paths between subject and object in the KG. MUKGE uses circular correlation as scoring function, and either applies the sigmoid or the bounded rectifier function as the function to obtain the triple confidence. Then the authors design the loss function to fit each positive triple to its corresponding confidence score. The authors assess their model on confidence prediction, relation fact ranking, and relation fact classification. For these three tasks, MUKGE outperforms the BEURRE and UKGE models, with a focus on asymmetrical relations. However, for all relation types, performance is competitive with those of other models, except for confidence prediction, where MUKGE is the best alternative.

## 6.2. Datasets with numerical values on edges

In the literature, five datasets that come from uncertain KBs are commonly used in UKG completion or confidence prediction tasks presented in the previous section [111]. **CN15K** is a subset of ConceptNet (presented in Section 3.2) where the numerical values correspond to the uncertainty of the triples [126]. The confidence scores for each triple are computed w.r.t. the number of sources and their reliability. **NL27K** is a subset of NELL dataset (presented in Section 4.1.2) where the confidence scores are computed and refined by an Expectation Maximization (EM) algorithm and a semi-supervised learning method. **PPI5K** is a KG that represents the protein-protein interactions,
where $\beta$ is a hyperparameter acting on the importance of the topological structure of the graph and $w$ is the numerical value on the edge. Then, the final function of FocusE is the following: $h(t) = \alpha g(t)$.

**ConFE [165]** encodes the tuples $(e, \tau)$ with $e$ an entity and $\tau$ an entity type by taking into account the uncertainty in each tuple. They consider entities and entity types as two different things in a KG and learn embeddings of entities and entity types in two distinct spaces with an asymmetric matrix to model their interactions. The scoring function is defined as $G(e, \tau) = e^{\top} M \tau$ where $M$ is the asymmetric matrix. And uncertainty is included within the loss function:

$$ \mathcal{L} = \sum_{(e, \tau) \in \mathcal{H}} \sum_{(e', \tau') \in \mathcal{H}'} \max(0, \gamma - G(e, \tau) + G(e', \tau')).C(e, \tau) $$

where $\mathcal{H}$ is the set of entities and their type and $\mathcal{H}'$ the set of corrupted tuples.

**CKRL [151]** introduces multiple levels of confidence, namely a local triple confidence, a global path confidence, a prior path confidence, and an adaptive path confidence. These confidence scores are integrated into the energy function following designed:

$$ E(T) = \sum_{(h,r,t) \in T} E(h, r, t) \cdot C(h, r, t) $$

where $E(h, r, t) = \|h + r - t\|$ and $C(h, r, t)$ the triple confidence score aggregating all levels of confidence.

**WaExt [77]** embeds triples of a KG by incorporating the weight $w$ associated to an edge $(h, r, t)$ in the scoring function in the following manner:

$$ f_w(h, r, t, w) = g(w) \cdot f(h, r, t) $$

and then minimizes a margin ranking loss function.

**Wang et al. [142]** model each entity and each relation as a multidimensional Gaussian distribution $N(\mu, \Sigma)$ where $\mu$ is a mean vector representing its position and $\Sigma$ is a diagonal covariance matrix representing its uncertainty.

**MUKGE** [91] aims to improve the generation of unseen facts for KGE training. The authors argue that PSL cannot take advantage of global multi-path information involving information loss to estimate the confidence of unseen facts. Indeed, PSL only considers information from simple logical rules with a path length of two as used in UKGE [20] (e.g., (college, synonym, university) ∧ (university, synonym, institute) → (college, synonym, institute)) and other paths in the graph between the subject and object of the inferred relation are not taken into account. To solve this issue, MUKGE introduces an algorithm called Uncertain ResourceRank to infer confidence scores for unseen triples based on the relevance of the entity pairs (subject, object). The relevance of an entity pair is computed w.r.t. the directed paths between subject and object in the KG. MUKGE uses circular correlation as scoring function, and either applies the sigmoid or the bounded rectifier function as the function to obtain the triple confidence. Then the authors design the loss function to fit each positive triple to its corresponding confidence score. The authors assess their model on confidence prediction, relation fact ranking, and relation fact classification. For these three tasks, MUKGE outperforms the BEURRE and UKGE models, with a focus on asymmetrical relations. However, for all relation types, performance is competitive with those of other models, except for confidence prediction, where MUKGE is the best alternative.

## 6.2. Datasets with numerical values on edges

In the literature, five datasets that come from uncertain KBs are commonly used in UKG completion or confidence prediction tasks presented in the previous section [111]. **CN15K** is a subset of ConceptNet (presented in Section 3.2) where the numerical values correspond to the uncertainty of the triples [126]. The confidence scores for each triple are computed w.r.t. the number of sources and their reliability. **NL27K** is a subset of NELL dataset (presented in Section 4.1.2) where the confidence scores are computed and refined by an Expectation Maximization (EM) algorithm and a semi-supervised learning method. **PPI5K** is a KG that represents the protein-protein interactions,
<table><caption>Table 1</caption><thead><tr><th>Model</th><th>Scoring function</th><th>Evaluation task</th><th>Dataset</th><th>Year</th></tr></thead><tbody><tr><td>IIKE [41]</td><td>TransE</td><td>Link prediction<br/>Triple classification</td><td>NELL<br/>FB15K (synthetic)</td><td>2016</td></tr><tr><td>URGE[62]</td><td>Matrix Factorization (proximity preservation)</td><td>Node Clustering<br/>Node Classification<br/>k-NN search</td><td>PPI<br/>DBLP</td><td>2017</td></tr><tr><td>CKRL [151]</td><td>TransE</td><td>KG Noise Detection<br/>KG Completion<br/>Triple Classification</td><td>FB15K (synthetic)</td><td>2018</td></tr><tr><td>GTransE [74]</td><td>TransE</td><td>KG Completion</td><td>FB15K-237 (synthetic)<br/>NELL (synthetic)</td><td>2019</td></tr><tr><td>UKGE [20]</td><td>DistMult</td><td>Confidence prediction<br/>Relation fact ranking<br/>Relation fact classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2019</td></tr><tr><td>UOKGE [11]</td><td>TransE</td><td>Confidence Prediction<br/>Triples Classification</td><td>CN15K</td><td>2019</td></tr><tr><td>SUKE [141]</td><td>DistMult</td><td>Link prediction<br/>Fact classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2021</td></tr><tr><td>BEURRE [21]</td><td>Gumbel boxes</td><td>Confidence prediction<br/>Relation fact ranking</td><td>CN15K<br/>NELL27k</td><td>2021</td></tr><tr><td>PASSLEAF [22]</td><td>RotatE<br/>ComplEx<br/>DistMult</td><td>Tail Entity Prediction<br/>Confidence Prediction</td><td>PPI5K<br/>NL27K<br/>CN15K<br/>WN18RR<br/>FB15K237</td><td>2021</td></tr><tr><td>GMUC [161]</td><td>Minimum Similarity</td><td>Link Prediction<br/>Confidence Prediction</td><td>NL27K</td><td>2021</td></tr><tr><td>ConfE [165]</td><td>RESCAL</td><td>Entity Type Noise Detection<br/>Entity Type Prediction</td><td>FB15kET (synthetic)<br/>YAGO43k (synthetic)</td><td>2021</td></tr><tr><td>FocusE [111]</td><td>Modifiable Scoring Layer</td><td>Link Prediction (High-Valued Links)</td><td>O*NET20K<br/>CN15K<br/>NL27K<br/>PPI5K</td><td>2021</td></tr><tr><td>WaExt [77]</td><td>TransE<br/>TransH<br/>DistMult<br/>ComplEx</td><td>Link Prediction<br/>Triple Classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2022</td></tr><tr><td>Wang et al. [142]</td><td>Similarity</td><td>Confidence Prediction<br/>Tail Entity Prediction</td><td>CN15K<br/>NL27K</td><td>2022</td></tr><tr><td>MUKGE [91]</td><td>Circular correlation</td><td>Confidence Prediction<br/>Relation Fact Ranking<br/>Relation Fact Classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2024</td></tr></tbody></table>

Uncertain embedding models taking into account at least one numeric value on edges with their scoring function, tasks handled, datasets on which they are evaluated, and the year of the publication.

and the numerical values correspond to the confidence relation [134]. O\*NET20K is a dataset introduced by [111] which includes descriptions about jobs and skills. The numerical values represent the strength of the relations. Some embedding models also generate their own synthetic noisy datasets with fictitious confidence scores following probability distributions.
<table><caption>Table 1</caption><thead><tr><th>Model</th><th>Scoring function</th><th>Evaluation task</th><th>Dataset</th><th>Year</th></tr></thead><tbody><tr><td>IIKE [41]</td><td>TransE</td><td>Link prediction<br/>Triple classification</td><td>NELL<br/>FB15K (synthetic)</td><td>2016</td></tr><tr><td>URGE[62]</td><td>Matrix Factorization (proximity preservation)</td><td>Node Clustering<br/>Node Classification<br/>k-NN search</td><td>PPI<br/>DBLP</td><td>2017</td></tr><tr><td>CKRL [151]</td><td>TransE</td><td>KG Noise Detection<br/>KG Completion<br/>Triple Classification</td><td>FB15K (synthetic)</td><td>2018</td></tr><tr><td>GTransE [74]</td><td>TransE</td><td>KG Completion</td><td>FB15K-237 (synthetic)<br/>NELL (synthetic)</td><td>2019</td></tr><tr><td>UKGE [20]</td><td>DistMult</td><td>Confidence prediction<br/>Relation fact ranking<br/>Relation fact classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2019</td></tr><tr><td>UOKGE [11]</td><td>TransE</td><td>Confidence Prediction<br/>Triples Classification</td><td>CN15K</td><td>2019</td></tr><tr><td>SUKE [141]</td><td>DistMult</td><td>Link prediction<br/>Fact classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2021</td></tr><tr><td>BEURRE [21]</td><td>Gumbel boxes</td><td>Confidence prediction<br/>Relation fact ranking</td><td>CN15K<br/>NELL27k</td><td>2021</td></tr><tr><td>PASSLEAF [22]</td><td>RotatE<br/>ComplEx<br/>DistMult</td><td>Tail Entity Prediction<br/>Confidence Prediction</td><td>PPI5K<br/>NL27K<br/>CN15K<br/>WN18RR<br/>FB15K237</td><td>2021</td></tr><tr><td>GMUC [161]</td><td>Minimum Similarity</td><td>Link Prediction<br/>Confidence Prediction</td><td>NL27K</td><td>2021</td></tr><tr><td>ConfE [165]</td><td>RESCAL</td><td>Entity Type Noise Detection<br/>Entity Type Prediction</td><td>FB15kET (synthetic)<br/>YAGO43k (synthetic)</td><td>2021</td></tr><tr><td>FocusE [111]</td><td>Modifiable Scoring Layer</td><td>Link Prediction (High-Valued Links)</td><td>O*NET20K<br/>CN15K<br/>NL27K<br/>PPI5K</td><td>2021</td></tr><tr><td>WaExt [77]</td><td>TransE<br/>TransH<br/>DistMult<br/>ComplEx</td><td>Link Prediction<br/>Triple Classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2022</td></tr><tr><td>Wang et al. [142]</td><td>Similarity</td><td>Confidence Prediction<br/>Tail Entity Prediction</td><td>CN15K<br/>NL27K</td><td>2022</td></tr><tr><td>MUKGE [91]</td><td>Circular correlation</td><td>Confidence Prediction<br/>Relation Fact Ranking<br/>Relation Fact Classification</td><td>CN15K<br/>NL27K<br/>PPI5K</td><td>2024</td></tr></tbody></table>

Uncertain embedding models taking into account at least one numeric value on edges with their scoring function, tasks handled, datasets on which they are evaluated, and the year of the publication.

and the numerical values correspond to the confidence relation [134]. O\*NET20K is a dataset introduced by [111] which includes descriptions about jobs and skills. The numerical values represent the strength of the relations. Some embedding models also generate their own synthetic noisy datasets with fictitious confidence scores following probability distributions.
## 7. Knowledge Alignment

Knowledge alignment, a.k.a. knowledge resolution or knowledge matching, is the process of finding relationships or correspondences between entities of different ontologies [39]. It is the first step in the pipeline after knowledge acquisition, which identifies candidate entities for knowledge fusion. For example, in Figure 8 the entity “Galaxy S23” of both graphs refers to the same entity in real world but are stemmed from two different sources. This task copes with the “redundancy” quality dimension (Section 4.2). Whether at instance level or at ontology level, many works tackle the knowledge alignment task. This section aims to provide an overview of the knowledge alignment task and approaches by gathering the various existing surveys [38, 42, 131].

The authors of [39] distinguish different types of matching that include semantic or syntactic approaches, e.g., string-based, language-based, subgraph-based, rule-based, embedding-based, or relational-based approaches. An example of such a rule-based method is the following, by [70]. For each relation $R_k$ over the two domains, define

$$R_k(a, b) \wedge \neg R_k(a', b') \Rightarrow a \neq a' \lor b \neq b' \qquad (1)$$

$$R_k(a, b) \wedge R_k(a', b') \Rightarrow a \equiv a' \wedge b \equiv b' \qquad (2)$$

$$\forall a, b \in o, a', b' \in o', \qquad (3)$$

where $R_k$ is a relation present in both graphs. To reduce complexity and avoid scalability issues, some approaches use blocking methods that avoid unnecessary comparisons by gathering entities. For example, Nguyen et al. [104] propose different strategies of blocking based on entities’ description such as token blocking, i.e., entities in the same cluster share at least one common token in their description, attribute clustering blocking, i.e., clusters the entities in the same group if their attributes are similar, and prefix-infix(-suffix) blocking, i.e., exploits the pattern in the description of the URI (e.g., URI infix) to create new blocks. After an optional blocking step, knowledge alignment methods are performed. Among them, [42] distinguish three methods: Sharing, Swapping, and Mapping. Sharing updates the entity embeddings produced by the embedding module according to the available similarity evidence of entities. Swapping updates the entity embeddings produced by the embedding module according to the available similarity evidence of entities but adds positive triples by leveraging aligned pairs e.g., $(h, h')$, $(t, t') \rightarrow (h', r, t) + (h, r, t')$. Mapping learns a linear transformation between the two embedding spaces of aligned KGs.

Furthermore, alignment approaches are diverse and varied, some of them leverage attributes of entities, or use only relations between entities where different depths of context (e.g., neighboring entities) are considered, while for other the path in the graph is an important aspect. We provide Table 2 that summarizes these different existing alignment approaches to get an overview strongly inspired by [42, 131].

The authors of [42] highlight that BERT_INT outperforms all models in terms of effectiveness and efficiency overall, especially when the KGs contain highly similar factual information. In fact, the alignment models that use language models such as BERT_INT are the most efficient for this task. The authors of [42] indicate the critical factors that affect the effectiveness of relation-based and attribute-based alignment methods, for instance:

- the depth of neighbors considered;

- negative sampling (for training), as the number of negatives are considered, the performances decrease;

- depending on inputs KGs to align, e.g., for OpenEA datasets, is not necessary to use attributes information, factual information is sufficient.

## 8. Uncertain Knowledge Fusion

In the previous section, we introduced the knowledge alignment task with a summary of the embedding-based approaches that tackle it. This task identifies equivalent entities and groups them into different clusters. The next step is the fusion of the attributes of the entities into the same clusters (as illustrated in Figure 7) since they may be redundant, inconsistent, contradictory, or of different granularity. We first define the task in Section 8.1, and we present the different approaches of fusion in Section 8.2.
## 7. Knowledge Alignment

Knowledge alignment, a.k.a. knowledge resolution or knowledge matching, is the process of finding relationships or correspondences between entities of different ontologies [39]. It is the first step in the pipeline after knowledge acquisition, which identifies candidate entities for knowledge fusion. For example, in Figure 8 the entity “Galaxy S23” of both graphs refers to the same entity in real world but are stemmed from two different sources. This task copes with the “redundancy” quality dimension (Section 4.2). Whether at instance level or at ontology level, many works tackle the knowledge alignment task. This section aims to provide an overview of the knowledge alignment task and approaches by gathering the various existing surveys [38, 42, 131].

The authors of [39] distinguish different types of matching that include semantic or syntactic approaches, e.g., string-based, language-based, subgraph-based, rule-based, embedding-based, or relational-based approaches. An example of such a rule-based method is the following, by [70]. For each relation $R_k$ over the two domains, define

$$R_k(a, b) \wedge \neg R_k(a', b') \Rightarrow a \neq a' \lor b \neq b' \qquad (1)$$

$$R_k(a, b) \wedge R_k(a', b') \Rightarrow a \equiv a' \wedge b \equiv b' \qquad (2)$$

$$\forall a, b \in o, a', b' \in o', \qquad (3)$$

where $R_k$ is a relation present in both graphs. To reduce complexity and avoid scalability issues, some approaches use blocking methods that avoid unnecessary comparisons by gathering entities. For example, Nguyen et al. [104] propose different strategies of blocking based on entities’ description such as token blocking, i.e., entities in the same cluster share at least one common token in their description, attribute clustering blocking, i.e., clusters the entities in the same group if their attributes are similar, and prefix-infix(-suffix) blocking, i.e., exploits the pattern in the description of the URI (e.g., URI infix) to create new blocks. After an optional blocking step, knowledge alignment methods are performed. Among them, [42] distinguish three methods: Sharing, Swapping, and Mapping. Sharing updates the entity embeddings produced by the embedding module according to the available similarity evidence of entities. Swapping updates the entity embeddings produced by the embedding module according to the available similarity evidence of entities but adds positive triples by leveraging aligned pairs e.g., $(h, h')$, $(t, t') \rightarrow (h', r, t) + (h, r, t')$. Mapping learns a linear transformation between the two embedding spaces of aligned KGs.

Furthermore, alignment approaches are diverse and varied, some of them leverage attributes of entities, or use only relations between entities where different depths of context (e.g., neighboring entities) are considered, while for other the path in the graph is an important aspect. We provide Table 2 that summarizes these different existing alignment approaches to get an overview strongly inspired by [42, 131].

The authors of [42] highlight that BERT_INT outperforms all models in terms of effectiveness and efficiency overall, especially when the KGs contain highly similar factual information. In fact, the alignment models that use language models such as BERT_INT are the most efficient for this task. The authors of [42] indicate the critical factors that affect the effectiveness of relation-based and attribute-based alignment methods, for instance:

- the depth of neighbors considered;

- negative sampling (for training), as the number of negatives are considered, the performances decrease;

- depending on inputs KGs to align, e.g., for OpenEA datasets, is not necessary to use attributes information, factual information is sufficient.

## 8. Uncertain Knowledge Fusion

In the previous section, we introduced the knowledge alignment task with a summary of the embedding-based approaches that tackle it. This task identifies equivalent entities and groups them into different clusters. The next step is the fusion of the attributes of the entities into the same clusters (as illustrated in Figure 7) since they may be redundant, inconsistent, contradictory, or of different granularity. We first define the task in Section 8.1, and we present the different approaches of fusion in Section 8.2.
<table><caption>Table 2
Recent embedding-based approaches that tackle knowledge alignment task.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Embedding</th><th rowspan="2">Method</th><th rowspan="2">Learning</th></tr><tr><th>One-hop</th><th>Multi-hop</th><th>Path</th></tr></thead><tbody><tr><td>MTransE [18]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>IPTransE [168]</td><td></td><td></td><td>•</td><td>Sharing</td><td>Semi-supervised</td></tr><tr><td>JAPE [129]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>BootEA [130]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Semi-supervised</td></tr><tr><td>KDCoE [19]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Semi-supervised</td></tr><tr><td>NTAM [85]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>GCNAlign [144]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>AttrE [136]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>IMUSE [54]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>SEA [80]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>RSN4EA [47]</td><td></td><td></td><td>•</td><td>Sharing</td><td>Supervised</td></tr><tr><td>GMNN [152]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>MuGNN [15]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>OTEA [115]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>NAEA [169]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>AVR-GCN [158]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>MultiKE [162]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>RDGCN [150]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>KECG [81]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>HGCN [149]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>MMEA [125]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>HMAN [155]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>AKE [89]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>RREA [94]</td><td></td><td>•</td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>BERT_INT [135]</td><td></td><td>•</td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>MTransE+RotatE [132]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr></tbody></table>

## 8.1. Task definition

The knowledge fusion step consists of studying how to combine various information about the same entity or concept from multiple data sources into a consistent and a unified one regarding the different deltas listed in the Section 5.1 [59, 108]. The authors of [31] identify three broad goals to be achieved for this challenging task:

- Completeness: measures the expected amount of data (number of tuples and number of attributes) at the output of the fusion task;

- Conciseness: measures the uniqueness of object representations in the integrated data (number of unique objects and number of unique attributes of objects);

- Correctness: measures the correctness of data, i.e., its conformity to the real world.

Therefore, data fusion corresponds to resolve conflicts from data with respect to these three goals. In [7], the authors distinguish two types of data conflicts from a data fusion perspective: *contradictions* and *uncertainties*. He defines *contradictions* as follows: “a contradiction is a conflict between two or more different non-null values that are all used to describe the same property of an object” and *uncertainties* as follows: “an uncertainty is a conflict
<table><caption>Table 2
Recent embedding-based approaches that tackle knowledge alignment task.</caption><thead><tr><th rowspan="2">Model</th><th colspan="3">Embedding</th><th rowspan="2">Method</th><th rowspan="2">Learning</th></tr><tr><th>One-hop</th><th>Multi-hop</th><th>Path</th></tr></thead><tbody><tr><td>MTransE [18]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>IPTransE [168]</td><td></td><td></td><td>•</td><td>Sharing</td><td>Semi-supervised</td></tr><tr><td>JAPE [129]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>BootEA [130]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Semi-supervised</td></tr><tr><td>KDCoE [19]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Semi-supervised</td></tr><tr><td>NTAM [85]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>GCNAlign [144]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>AttrE [136]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>IMUSE [54]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>SEA [80]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>RSN4EA [47]</td><td></td><td></td><td>•</td><td>Sharing</td><td>Supervised</td></tr><tr><td>GMNN [152]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>MuGNN [15]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>OTEA [115]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>NAEA [169]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>AVR-GCN [158]</td><td></td><td>•</td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>MultiKE [162]</td><td>•</td><td></td><td></td><td>Swapping</td><td>Supervised</td></tr><tr><td>RDGCN [150]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>KECG [81]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>HGCN [149]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>MMEA [125]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>HMAN [155]</td><td></td><td>•</td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>AKE [89]</td><td>•</td><td></td><td></td><td>Mapping</td><td>Supervised</td></tr><tr><td>RREA [94]</td><td></td><td>•</td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>BERT_INT [135]</td><td></td><td>•</td><td></td><td>Sharing</td><td>Supervised</td></tr><tr><td>MTransE+RotatE [132]</td><td>•</td><td></td><td></td><td>Sharing</td><td>Supervised</td></tr></tbody></table>

## 8.1. Task definition

The knowledge fusion step consists of studying how to combine various information about the same entity or concept from multiple data sources into a consistent and a unified one regarding the different deltas listed in the Section 5.1 [59, 108]. The authors of [31] identify three broad goals to be achieved for this challenging task:

- Completeness: measures the expected amount of data (number of tuples and number of attributes) at the output of the fusion task;

- Conciseness: measures the uniqueness of object representations in the integrated data (number of unique objects and number of unique attributes of objects);

- Correctness: measures the correctness of data, i.e., its conformity to the real world.

Therefore, data fusion corresponds to resolve conflicts from data with respect to these three goals. In [7], the authors distinguish two types of data conflicts from a data fusion perspective: *contradictions* and *uncertainties*. He defines *contradictions* as follows: “a contradiction is a conflict between two or more different non-null values that are all used to describe the same property of an object” and *uncertainties* as follows: “an uncertainty is a conflict
between a non-null value and one or more null values that are all used to describe the same property of an object". We adopt the same definition of *contradictions*, but we do not consider the same definition of *uncertainty*. We define the second type of conflict as a difference in granularity of knowledge, as illustrated in Figure 6.

In this survey, “uncertainty” is a more generic term whose sources lie in knowledge deltas and the inaccuracy of each step in the knowledge integration pipeline, including knowledge acquisition. Indeed, when we integrate knowledge from several sources, the quality of the information varies, and we need to determine the trustworthy information by performing a Truth Inferring (TI) task.

According to Rekatsinas [117] different TI strategies can be adopted. There are simple strategies that estimate the true values of the entities compared to the other values provided by the sources by applying a majority vote or an average on them. Then, there are strategies that use the trustworthiness of the sources to quantify the true values of the objects, and it is even possible to establish a precision for each class of object and for each source. The problem with simple strategies is that they do not take into account the varied quality of data sources [84], but these are often used to initialize true values to start iterative TI methods.

For example, we consider that we have previously extracted knowledge in triple form (subject, predicate, object) about the mobile phone “Galaxy S23” from several sources $S_1, S_2, ..., S_n$ resulting in the table at the bottom of Figure 8 where entities have already been aligned. We also represent the table as a graph for sources $S_1$ and $S_2$. Several papers on data fusion use the term “data item”, i.e., (entity, attribute, value) instead of the term “triple”

<figure><img src="image_11.png" /><figcaption>Fig. 8. Extracted triples represented as graphs from two sources $S_1$ and $S_2$. $S_1$ on the left and $S_2$ on the right. With different types of conflicts: contradictory information (◆), duplicate (■), and difference of granularity (●).</figcaption></figure>

to refer to an element to be merged. However, in practice a data item is equivalent to a triple (subject, predicate, object). Each row of the table corresponds to an entity of a graph and its attributes, for example the entity $e_1$ corresponds to the node “Galaxy S23” of the graph and the values associated with $e_1$ in the table are the objects of
between a non-null value and one or more null values that are all used to describe the same property of an object". We adopt the same definition of *contradictions*, but we do not consider the same definition of *uncertainty*. We define the second type of conflict as a difference in granularity of knowledge, as illustrated in Figure 6.

In this survey, “uncertainty” is a more generic term whose sources lie in knowledge deltas and the inaccuracy of each step in the knowledge integration pipeline, including knowledge acquisition. Indeed, when we integrate knowledge from several sources, the quality of the information varies, and we need to determine the trustworthy information by performing a Truth Inferring (TI) task.

According to Rekatsinas [117] different TI strategies can be adopted. There are simple strategies that estimate the true values of the entities compared to the other values provided by the sources by applying a majority vote or an average on them. Then, there are strategies that use the trustworthiness of the sources to quantify the true values of the objects, and it is even possible to establish a precision for each class of object and for each source. The problem with simple strategies is that they do not take into account the varied quality of data sources [84], but these are often used to initialize true values to start iterative TI methods.

For example, we consider that we have previously extracted knowledge in triple form (subject, predicate, object) about the mobile phone “Galaxy S23” from several sources $S_1, S_2, ..., S_n$ resulting in the table at the bottom of Figure 8 where entities have already been aligned. We also represent the table as a graph for sources $S_1$ and $S_2$. Several papers on data fusion use the term “data item”, i.e., (entity, attribute, value) instead of the term “triple”

<figure><img src="image_10.png" /><figcaption>Fig. 8. Extracted triples represented as graphs from two sources $S_1$ and $S_2$. $S_1$ on the left and $S_2$ on the right. With different types of conflicts: contradictory information (◆), duplicate (■), and difference of granularity (●).</figcaption></figure>

to refer to an element to be merged. However, in practice a data item is equivalent to a triple (subject, predicate, object). Each row of the table corresponds to an entity of a graph and its attributes, for example the entity $e_1$ corresponds to the node “Galaxy S23” of the graph and the values associated with $e_1$ in the table are the objects of
the triples. These objects are linked to the subject "Galaxy S23" by the predicates identified by column headers, as depicted in Figure 8. The data extracted from both sources is almost the same except for the relations *software* and *inception* where differences of granularity appear (e.g., the price of the mobile phone). Source $S_3$ states that the brand of the mobile phone is "Apple", contradicting the first two data sources. Another example of contradiction is the invalidity of the phone's inception date provided by $S_1$. We can also distinguish two different levels of granularity. The first level concerns literals (i.e., numerical values, strings, etc.), for example the granularity of a product description as depicted in Figure 8. The second level concerns concepts e.g., Korea vs. Asia to indicate the location of Suwon-si as depicted in Figure 10. On top of that, different scales of knowledge are possible as illustrated in Figure 8 among triples extracted from $S_1$ we have (Galaxy S23, hasPrice, "800€") and from source $S_2$ we have (Galaxy S23, hasPrice, "expensive"), the terms "800€" and "expensive" are indeed not in the same scale, one indicates the exact price while the second one gives a qualification of the price. In all cases, the first and second sources provide complementary pieces of information for other attributes. Figure 9 shows the resulting graph after the reconciliation step that includes the knowledge alignment and the fusion step where the most complete representation of the Galaxy S23 entity is produced.

<figure><img src="image_12.png" /><figcaption>Fig. 9. Resulting KG after reconciliation step (alignment and fusion).</figcaption></figure>

In the next section, we survey several methods that address knowledge fusion for truth discovery. We will use the terms "truth inferring", "truth finding" or "truth discovery" in the same way.

## 8.2. Fusion Approaches

Fusion methods are identified in Table 3, which provides an overview and indicates if the methods can handle some characteristics of data or data sources such as numerical data, data granularity, and dependency between data sources.

**SLIMFAST** [117] leverages knowledge domain features to improve the quality estimation of data sources. For example, if the data is extracted from scientific articles, the authors suggest using features such as the number of citations to the article or the year of publication, which can influence the quality of the source. Domain-specific features are included in the parameters of the logistic function that estimates source quality. To merge the data, they apply statistical learning to estimate source quality, then apply probabilistic inference to predict true values. To estimate the parameters, they use either the EM algorithm if the user does not provide labeled ground truth data, or the Empirical Risk Minimization algorithm if the user does.

**ACCU** [32] includes the interdependence between data sources in the truth discovery process. The intuition is that it is possible that a single source could provide the true value and that all the other sources could provide false
the triples. These objects are linked to the subject "Galaxy S23" by the predicates identified by column headers, as depicted in Figure 8. The data extracted from both sources is almost the same except for the relations *software* and *inception* where differences of granularity appear (e.g., the price of the mobile phone). Source $S_3$ states that the brand of the mobile phone is "Apple", contradicting the first two data sources. Another example of contradiction is the invalidity of the phone's inception date provided by $S_1$. We can also distinguish two different levels of granularity. The first level concerns literals (i.e., numerical values, strings, etc.), for example the granularity of a product description as depicted in Figure 8. The second level concerns concepts e.g., Korea vs. Asia to indicate the location of Suwon-si as depicted in Figure 10. On top of that, different scales of knowledge are possible as illustrated in Figure 8 among triples extracted from $S_1$ we have (Galaxy S23, hasPrice, "800€") and from source $S_2$ we have (Galaxy S23, hasPrice, "expensive"), the terms "800€" and "expensive" are indeed not in the same scale, one indicates the exact price while the second one gives a qualification of the price. In all cases, the first and second sources provide complementary pieces of information for other attributes. Figure 9 shows the resulting graph after the reconciliation step that includes the knowledge alignment and the fusion step where the most complete representation of the Galaxy S23 entity is produced.

<figure><img src="image_11.png" /><figcaption>Fig. 9. Resulting KG after reconciliation step (alignment and fusion).</figcaption></figure>

In the next section, we survey several methods that address knowledge fusion for truth discovery. We will use the terms "truth inferring", "truth finding" or "truth discovery" in the same way.

## 8.2. Fusion Approaches

Fusion methods are identified in Table 3, which provides an overview and indicates if the methods can handle some characteristics of data or data sources such as numerical data, data granularity, and dependency between data sources.

**SLIMFAST** [117] leverages knowledge domain features to improve the quality estimation of data sources. For example, if the data is extracted from scientific articles, the authors suggest using features such as the number of citations to the article or the year of publication, which can influence the quality of the source. Domain-specific features are included in the parameters of the logistic function that estimates source quality. To merge the data, they apply statistical learning to estimate source quality, then apply probabilistic inference to predict true values. To estimate the parameters, they use either the EM algorithm if the user does not provide labeled ground truth data, or the Empirical Risk Minimization algorithm if the user does.

**ACCU** [32] includes the interdependence between data sources in the truth discovery process. The intuition is that it is possible that a single source could provide the true value and that all the other sources could provide false
<table><caption>Table 3: Description of the tasks and characteristics of the data or sources that are (●) or not (○) addressed by the knowledge fusion models. (TI) stands for Truth Inference, (SQ) stands for Source Quality.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Task</th><th rowspan="2">Modeling</th><th colspan="3">Awareness</th><th rowspan="2">Datasets</th><th rowspan="2">Year</th></tr><tr><th>TI</th><th>SQ</th><th>Numerical Data</th><th>Granularity</th><th>Source dependence</th></tr></thead><tbody><tr><td>GTM [163]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Wikipedia edit history of city population</td><td>2012</td></tr><tr><td>LTM [164]</td><td>•</td><td>•</td><td>Probabilistic</td><td>○</td><td>○</td><td>○</td><td>People biographies<br>Book author<br>Movie director<br>Synthetic<br>Books<br>Population<br>Stocks</td><td>2012</td></tr><tr><td>LCA [113]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Fantasy SCOTUS<br>Population<br>Stocks</td><td>2013</td></tr><tr><td>KBT [34]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Triples collected by Knowledge Vault<br>Synthetic<br>Stocks</td><td>2015</td></tr><tr><td>SLIMFAST [117]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Demonstrations<br>Crowds<br>Genomics</td><td>2017</td></tr><tr><td>LFC [116]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Digital mammography<br>Breast MRI<br>Item Compare<br>4-Domain<br>Yahoo QA</td><td>2010</td></tr><tr><td>DOCS [166]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>SFV<br>Dataset created from baobaozhido (crowdsourcing platform)<br>Books (from AbeBooks.com)<br>Flight</td><td>2017<br>2012</td></tr><tr><td>MDC [88]</td><td>•</td><td>•</td><td>Representation Learning</td><td>○</td><td>○</td><td>○</td><td>Synthetic<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2009<br>2014</td></tr><tr><td>POPACCU [33]</td><td>○</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Stocks<br>Flight<br>Stock 1<br>Stock 2<br>Weather<br>Address<br>Book authors<br>Synthetic<br>People biographies<br>BirthPlaces<br>Heritages<br>Dataset built from [14]</td><td>2007<br>2016</td></tr><tr><td>ACCU [32]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>•</td><td>Subgraph of Freebase as prior knowledge (KG)<br>Synthetic<br>SFV</td><td>2020<br>2015</td></tr><tr><td>CRH [84]</td><td>•</td><td>•</td><td>Loss Optimization</td><td>•</td><td>○</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr><tr><td>Record Fusion [55]</td><td>•</td><td>○</td><td>Softmax Classifier</td><td>•</td><td>○</td><td>○</td><td>Indoor Floorplan<br>Synthetic<br>Population</td><td>2016</td></tr><tr><td>TruthFinder [159]</td><td>•</td><td>•</td><td>Probabilistic</td><td>○</td><td>•</td><td>○</td><td>Book authors<br>Synthetic<br>People biographies<br>BirthPlaces<br>Heritages<br>Dataset built from [14]</td><td>2007<br>2016</td></tr><tr><td>ASUMS [5]</td><td>•</td><td>•</td><td>Belief functions</td><td>○</td><td>•</td><td>○</td><td>Subgraph of Freebase as prior knowledge (KG)<br>Synthetic<br>SFV</td><td>2020<br>2015</td></tr><tr><td>TDH [72]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr><tr><td>TKGC [63]</td><td>•</td><td>•</td><td>Representation Learning</td><td>•</td><td>•</td><td>○</td><td>Subgraph of Freebase as prior knowledge (KG)<br>Synthetic<br>SFV</td><td>2022<br>2020</td></tr><tr><td>OKELE [14]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Synthetic<br>SFV</td><td>2020<br>2015</td></tr><tr><td>FairCrowd [92]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr><tr><td>HYBRID [82]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr><tr><td>CATD [83]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr><tr><td>KDEm [140]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Dataset from crowdsourcing platform<br>Book<br>Synthetic<br>City Population<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014</td></tr></tbody></table>
<table><caption>Table 3: Description of the tasks and characteristics of the data or sources that are (●) or not (○) addressed by the knowledge fusion models. (TI) stands for Truth Inference, (SQ) stands for Source Quality.</caption><thead><tr><th rowspan="2">Model</th><th colspan="2">Task</th><th rowspan="2">Modeling</th><th colspan="3">Awareness</th><th rowspan="2">Datasets</th><th rowspan="2">Year</th></tr><tr><th>TI</th><th>SQ</th><th>Numerical Data</th><th>Granularity</th><th>Source dependence</th></tr></thead><tbody><tr><td>GTM [163]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Wikipedia edit history of city population</td><td>2012</td></tr><tr><td>LTM [164]</td><td>•</td><td>•</td><td>Probabilistic</td><td>○</td><td>○</td><td>○</td><td>People biographies<br>Book author<br>Movie director<br>Synthetic<br>Books<br>Population<br>Stocks</td><td>2012</td></tr><tr><td>LCA [113]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Fantasy SCOTUS<br>Population<br>Stocks</td><td>2013</td></tr><tr><td>KBT [34]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Triples collected by Knowledge Vault<br>Synthetic<br>Stocks</td><td>2015</td></tr><tr><td>SLIMFAST [117]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Demonstrations<br>Crowds<br>Genomics</td><td>2017</td></tr><tr><td>LFC [116]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Digital mammography<br>Breast MRI<br>Item Compare<br>4-Domain<br>Yahoo QA</td><td>2010</td></tr><tr><td>DOCS [166]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>SFV<br>Dataset created from baobaozhido (crowdsourcing platform)<br>Books (from AbeBooks.com)<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>MDC [88]</td><td>•</td><td>•</td><td>Representation Learning</td><td>○</td><td>○</td><td>○</td><td>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>POPACCU [33]</td><td>○</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>ACCU [32]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>•</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>CRH [84]</td><td>•</td><td>•</td><td>Loss Optimization</td><td>•</td><td>○</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>Record Fusion [55]</td><td>•</td><td>○</td><td>Softmax Classifier</td><td>•</td><td>○</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2012<br>2009<br>2014</td></tr><tr><td>TruthFinder [159]</td><td>•</td><td>•</td><td>Probabilistic</td><td>○</td><td>•</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2007<br>2016<br>2019<br>2017</td></tr><tr><td>ASUMS [5]</td><td>•</td><td>•</td><td>Belief functions</td><td>○</td><td>•</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2007<br>2016<br>2019<br>2017</td></tr><tr><td>TDH [72]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2016<br>2019<br>2017</td></tr><tr><td>TKGC [63]</td><td>•</td><td>•</td><td>Representation Learning</td><td>•</td><td>•</td><td>○</td><td>Stocks<br>Flight<br>Synthetic<br>Synthetic<br>Weather Forecast (from Wunderground, HAM weather, World Weather Online)</td><td>2017<br>2016<br>2019<br>2017</td></tr><tr><td>OKELE [14]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Synthetic<br>Synthetic<br>Synthetic<br>SFV</td><td>2020<br>2015</td></tr><tr><td>FairCrowd [92]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Synthetic<br>Synthetic<br>Synthetic<br>SFV</td><td>2020<br>2015</td></tr><tr><td>HYBRID [82]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Synthetic<br>Synthetic<br>Synthetic<br>Synthetic<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014<br>2017<br>2014</td></tr><tr><td>CATD [83]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>○</td><td>○</td><td>Synthetic<br>Synthetic<br>Biography<br>Indoor Floorplan<br>Synthetic<br>Population</td><td>2017<br>2014<br>2017<br>2014</td></tr><tr><td>KDEm [140]</td><td>•</td><td>•</td><td>Probabilistic</td><td>•</td><td>•</td><td>○</td><td>Synthetic<br>Synthetic<br>Population</td><td>2016</td></tr></tbody></table>
values knowing that some of them can copy on each other and therefore, spread false values. Thus, if a data source provides a value different from all the others, it is not systematically false. They define the dependency between two sources if there is a part of their data that comes directly or transitively from a common source, and it is computed by Bayesian models. Then, to discover the true value, they combine this dependency evaluation and the accuracy of the data sources which are computed in relation to the confidences of the values and dependencies between sources.

**POPACCU** [33], unlike ACCU, considers that the data sources are independent and that only one value can be correct.

**CRH** [84] (Conflict Resolution on Heterogeneous Data) estimates the reliability of a source, it uses all types of data simultaneously instead of focusing on a single one, but requires availability of all data for each entity. To initiate the source reliability score, it first applies a simple conflict resolution method, such as majority or average voting.

**MDC** [88] takes into account the semantic aspect of values. To illustrate the importance of semantics, in the paper the authors provide the following example: a first data source provides the true value “common cold”, another source claims the value is “sinus infection” while a last source claims the value is “bone fracture”, instead of examining all values at the same level, MDC calculates semantic proximity among values. This semantic proximity calculation allows evaluating how close a value is to the true value. The semantics of the values are captured by their vector representations learned by following the idea that if two values share similar words, then their vectors should be similar.

**DOCS** [166] is a system deployed on the Amazon Mechanical Turk that takes into account the precision of the answers of each worker to assign a specific task from a specific domain to the right worker. Regarding true value inference, the system takes advantage of the inherent relationships between the reliability of workers (which can be seen as data sources) and the true value. Thus, it considers two events: (1) let $v$ be the value of an entity provided by a source $s$, if the quality of the values provided by $s$ of entities in the same domain as $v$ is high then $v$ is likely to be correct; (2) then, if a source $s$ often provides correct values for a domain $d$, then $s$ has high reliability for domain $d$.

**TruthFinder** [159] claims that a fact is more likely to be true if it is provided by a reliable source, and that a source is reliable if it provides proven facts. Thus, an interdependence between facts and sources appears and consequently TruthFinder uses three elements for its iterative trust discovery process: the trustworthiness of sources, the confidence of facts, and the influences between facts. The trustworthiness $t(w)$ of a source $w$ is computed by

$$t(w) = \frac{\sum_{f \in F(w)} s(t)}{|F(w)|},$$

and the confidence $s(t)$ of a fact $t$ is computed by

$$s(t) = 1 - \prod_{w \in W(t)} (1 - t(w)).$$

Then, they use the logarithm to facilitate the computation and to obtain the final scores. Although simple, these definitions integrate the influence of facts and take into account the dependence between data sources.

While most existing methods consider more generic values of a correct one as false, **TDH** [72] leverages the hierarchical structure of knowledge (i.e., its one aspect of the granularity) to apply the fusion of data extracted from different sources. The idea behind is that multiple values in the hierarchy of an entity could be correct even if the predicate is functional. For example, in Figure 10, “Korea”, and “Asia” are correct values for the location of “Suwon” even though one of both values is more specific. Such a modeling should not negatively impact the assessment of the reliability of the source. Instead of evaluating the value as *correct* or *incorrect*, they consider three classifications, namely *exactly correct*, *hierarchically correct*, and *incorrect*. Therefore, each source has its own generalization tendency and reliability.

In a same way, **ASUMS** [5] adapts existing truth discovery models by considering that not all values are necessarily conflicting and identifies a partial order between the values of an attribute using the “subClassOf” and “partOf” relationships. To do this, they use belief functions capable of modeling ignorance and uncertainty and allowing the incorporation of knowledge about the relations between values. Thus, in this modeling several true values can
values knowing that some of them can copy on each other and therefore, spread false values. Thus, if a data source provides a value different from all the others, it is not systematically false. They define the dependency between two sources if there is a part of their data that comes directly or transitively from a common source, and it is computed by Bayesian models. Then, to discover the true value, they combine this dependency evaluation and the accuracy of the data sources which are computed in relation to the confidences of the values and dependencies between sources.

**POPACCU** [33], unlike ACCU, considers that the data sources are independent and that only one value can be correct.

**CRH** [84] (Conflict Resolution on Heterogeneous Data) estimates the reliability of a source, it uses all types of data simultaneously instead of focusing on a single one, but requires availability of all data for each entity. To initiate the source reliability score, it first applies a simple conflict resolution method, such as majority or average voting.

**MDC** [88] takes into account the semantic aspect of values. To illustrate the importance of semantics, in the paper the authors provide the following example: a first data source provides the true value “common cold”, another source claims the value is “sinus infection” while a last source claims the value is “bone fracture”, instead of examining all values at the same level, MDC calculates semantic proximity among values. This semantic proximity calculation allows evaluating how close a value is to the true value. The semantics of the values are captured by their vector representations learned by following the idea that if two values share similar words, then their vectors should be similar.

**DOCS** [166] is a system deployed on the Amazon Mechanical Turk that takes into account the precision of the answers of each worker to assign a specific task from a specific domain to the right worker. Regarding true value inference, the system takes advantage of the inherent relationships between the reliability of workers (which can be seen as data sources) and the true value. Thus, it considers two events: (1) let $v$ be the value of an entity provided by a source $s$, if the quality of the values provided by $s$ of entities in the same domain as $v$ is high then $v$ is likely to be correct; (2) then, if a source $s$ often provides correct values for a domain $d$, then $s$ has high reliability for domain $d$.

**TruthFinder** [159] claims that a fact is more likely to be true if it is provided by a reliable source, and that a source is reliable if it provides proven facts. Thus, an interdependence between facts and sources appears and consequently TruthFinder uses three elements for its iterative trust discovery process: the trustworthiness of sources, the confidence of facts, and the influences between facts. The trustworthiness $t(w)$ of a source $w$ is computed by

$$t(w) = \frac{\sum_{f \in F(w)} s(t)}{|F(w)|},$$

and the confidence $s(t)$ of a fact $t$ is computed by

$$s(t) = 1 - \prod_{w \in W(t)} (1 - t(w)).$$

Then, they use the logarithm to facilitate the computation and to obtain the final scores. Although simple, these definitions integrate the influence of facts and take into account the dependence between data sources.

While most existing methods consider more generic values of a correct one as false, **TDH** [72] leverages the hierarchical structure of knowledge (i.e., its one aspect of the granularity) to apply the fusion of data extracted from different sources. The idea behind is that multiple values in the hierarchy of an entity could be correct even if the predicate is functional. For example, in Figure 10, “Korea”, and “Asia” are correct values for the location of “Suwon” even though one of both values is more specific. Such a modeling should not negatively impact the assessment of the reliability of the source. Instead of evaluating the value as *correct* or *incorrect*, they consider three classifications, namely *exactly correct*, *hierarchically correct*, and *incorrect*. Therefore, each source has its own generalization tendency and reliability.

In a same way, **ASUMS** [5] adapts existing truth discovery models by considering that not all values are necessarily conflicting and identifies a partial order between the values of an attribute using the “subClassOf” and “partOf” relationships. To do this, they use belief functions capable of modeling ignorance and uncertainty and allowing the incorporation of knowledge about the relations between values. Thus, in this modeling several true values can
coexist but of different granularity. Therefore, all facts more generic than a certain fact are true and conflicting facts are facts located at the same hierarchical level but with different values.

<figure><img src="image_13.png" /><figcaption>Fig. 10. Illustration of a partial ordering.</figcaption></figure>

**LFC** [116] also measures the performance of each source (e.g., annotators in the paper) by their specificity and sensitivity relating to the unknown gold standard dataset to give higher weights to the best-performing sources. The estimations are iteratively performed via the EM algorithm, where the missing data is the true value initialized by applying a majority voting. Specificity and sensitivity are then estimated in turn, and so on until convergence is reached.

**LCA** [113] includes four models with different sophistication, namely: SimpleLCA, GuessLCA, MistakeLCA, LieLCA. LCA is a probabilistic model where the true value is a multinomial latent variable. To infer the truth, an EM algorithm is used to compute the trustworthiness of each source with respect to the claims it makes, then the true value is computed based on the trustworthiness of the sources.

**KBT** [34] focuses on assessing the quality of Web sources from which facts are extracted and also evaluated. Facts are extracted as triples (subject, predicate, object) from Web pages using Knowledge Vault, which is composed of 16 different extractors. The authors extend the ACCU model by improving the estimation of the reliability of a source by distinguishing the error that comes from the fact against the error that comes from the extraction of the fact. However, they do not consider granularity or the fact that several correct values may coexist for a data item.

While other approaches are focused on categorical data, **GTM** [163] (Gaussian Truth Model) tackles the truth finding task on numerical data. This model focuses on the position of numerical claim values $v_c$ relative to others in terms of distance to find the truth. To embed the notion of distance in their model, the authors consider the truth of each entity as a random variable and use it as the mean parameter in the probabilistic distribution for each claimed value of the observed entities. To do this, they leverage a Gaussian distribution for its ability to model errors thanks to its quadratic penalty. Regarding the evaluation of the quality of data sources, they assume that quality is related to the closeness of the claims to the truth. Therefore, the quality of a source is modeled by the variance of the Gaussian distribution, e.g., a high-quality source is represented by a low variance. As aforementioned, the model can take as input the output of another truth finding method or a basic truth estimate (e.g., mean or median value) to limit the effect of possible outliers on the maximum likelihood estimate (MLE). The quality of each source is determined from a prior inverse Gamma distribution, and the truth for an entity is determined from a prior Gaussian distribution. Then, to compute the truth and source quality, they perform an EM algorithm.

In the same manner as **GTM**, **LTM** [164] incorporates prior knowledge about sources or truth into the truth finding process and introduces the notion of two-sided source quality. It simultaneously deduces the quality of the source and the truth, as both influence each other. To compute the quality of data sources, the authors consider each source as a classifier, with its own confusion matrix. Thus, the quality of a source is defined by its sensitivity (or
coexist but of different granularity. Therefore, all facts more generic than a certain fact are true and conflicting facts
are facts located at the same hierarchical level but with different values.

<figure><img src="image_12.png" /><figcaption>Fig. 10. Illustration of a partial ordering.</figcaption></figure>

**LFC** [116] also measures the performance of each source (e.g., annotators in the paper) by their specificity and sensitivity relating to the unknown gold standard dataset to give higher weights to the best-performing sources. The estimations are iteratively performed via the EM algorithm, where the missing data is the true value initialized by applying a majority voting. Specificity and sensitivity are then estimated in turn, and so on until convergence is reached.

**LCA** [113] includes four models with different sophistication, namely: SimpleLCA, GuessLCA, MistakeLCA, LieLCA. LCA is a probabilistic model where the true value is a multinomial latent variable. To infer the truth, an EM algorithm is used to compute the trustworthiness of each source with respect to the claims it makes, then the true value is computed based on the trustworthiness of the sources.

**KBT** [34] focuses on assessing the quality of Web sources from which facts are extracted and also evaluated. Facts are extracted as triples (subject, predicate, object) from Web pages using Knowledge Vault, which is composed of 16 different extractors. The authors extend the ACCU model by improving the estimation of the reliability of a source by distinguishing the error that comes from the fact against the error that comes from the extraction of the fact. However, they do not consider granularity or the fact that several correct values may coexist for a data item.

While other approaches are focused on categorical data, **GTM** [163] (Gaussian Truth Model) tackles the truth finding task on numerical data. This model focuses on the position of numerical claim values $v_c$ relative to others in terms of distance to find the truth. To embed the notion of distance in their model, the authors consider the truth of each entity as a random variable and use it as the mean parameter in the probabilistic distribution for each claimed value of the observed entities. To do this, they leverage a Gaussian distribution for its ability to model errors thanks to its quadratic penalty. Regarding the evaluation of the quality of data sources, they assume that quality is related to the closeness of the claims to the truth. Therefore, the quality of a source is modeled by the variance of the Gaussian distribution, e.g., a high-quality source is represented by a low variance. As aforementioned, the model can take as input the output of another truth finding method or a basic truth estimate (e.g., mean or median value) to limit the effect of possible outliers on the maximum likelihood estimate (MLE). The quality of each source is determined from a prior inverse Gamma distribution, and the truth for an entity is determined from a prior Gaussian distribution. Then, to compute the truth and source quality, they perform an EM algorithm.

In the same manner as **GTM**, **LTM** [164] incorporates prior knowledge about sources or truth into the truth finding process and introduces the notion of two-sided source quality. It simultaneously deduces the quality of the source and the truth, as both influence each other. To compute the quality of data sources, the authors consider each source as a classifier, with its own confusion matrix. Thus, the quality of a source is defined by its sensitivity (or
recall) which corresponds to the “false negative rate” and by its specificity which corresponds to the “false positive
rate” that are two independent measures. Both sensitivity and specificity are generated from a Beta distribution: the
parameters for specificity are “the prior false positive count” and “the prior true negative count” and the parameters
for sensitivity are “the prior true positive count” and “the prior false negative count”. The prior truth probability is
also modeled by a Beta distribution with the parameters “the prior true count” and “the prior false count” for each
distinct (entity, value) pair. The truth value is generated by a Bernoulli distribution with a parameter $\theta$, corresponding
to the prior probability that the value is true. Finally, the truth and the quality of sources are inferred by a Collapsed
Gibbs Sampling.

**Record Fusion** [55] merges knowledge by relying on integrity constraints, quantitative statistics, and provenance information if this latter is available. To find the true value of each table cell, they use one classifier per column (attribute) present in the table (dataset). These softmax classifiers can be modular, for example a logistic regression, a decision tree, a neural network, and so on. Three representation models are explored to create the feature vector, which will then be provided to the classifier. The first representation is the role of a cell at the column level, where three different strategies are proposed: the first acts on the format of the data (e.g., the letters of the alphabet are replaced by the token “A”, numbers by “N” and characters (e.g., “”, “:”, “”, or space) by “S” and they get a vector from a n-gram model), the second strategy is to cluster attribute values, then the last strategy consider a matrix of embeddings which will map all the values of a column (attribute) in a Euclidean space and for each cell, computes the distance between its position in space and the average position of the other values. The second concerns the role of the cell at the row level (tuple), i.e., it captures the relationship of the attribute with other attributes in its row (entity). Two signals are leveraged, the first one includes the counts of pairs of attributes that are seen together, and the second one captures how often a cell occurs among rows within its own entity. Finally, the third concerns the role of the cell in relation to the complete dataset (table), i.e., takes into consideration the number of denial constraint violations, includes the source information only if available since entities can have different provenance and each source can have different levels of trust. Then, the last step consists in training the different classifiers by a stage-wise additive model for a number of *T* iterations: (1) they learn the softmax classifiers with the original dataset, (2) use previous predictions to construct a new dynamic feature, (3) and learn again the classifiers using these new sets of features. For cells where the label is unknown, they assign a majority vote as their weak labels. They obtain good performance on datasets Flight, Stock, Weather, and Address about 94%-98% of precision.

In contrast to many fusion truth inferring approaches, FaitCrowd [92] measures the quality of data sources over
several degrees, with one quality degree for each knowledge topic for a crowdsourcing case. FaitCrowd represents
the expertise of each source for each topic by a Gaussian distribution. Then, it models the true value provided by
a source for a certain question on a given topic as a logistic function depending on the contribution ratio of the
source on the topic, the expertise of the source and a bias. To estimate the parameters, the model uses the Gibbs-EM
inference method that alternates between Gibbs sampling and gradient descent.

**TKGC** [63] takes advantage of prior knowledge from the KG when feeding it and considers that the noise affect-
ing the truth is represented by a probability distribution determined by the data source. To estimate the difference
between the truth value and the value supplied by a source, the authors use a difference function adapted to each
data type, i.e., categorical data, numerical or datetime value, and string. This difference function takes as input the
representation vectors previously learned in a fact scoring setting for KG completion. In fact, the probability of this
latter function follows a Gaussian distribution $N(0, (k_a\sigma_s^2))$ where $k_a$ is a regularization factor and $\sigma_s$ representing
the noise of a data source. Then the truth inference is performed through a semi-supervised algorithm.

**OKELE** [14] models the probability of a fact being true by a latent random variable following a Beta($\beta_0, \beta_1$)
distribution, with $\beta_1$ corresponding to prior true count of the fact and $\beta_0$ corresponding to its prior false count. The
quality of a data source is represented by its error variance $\omega_s$ that follows a scaled inverse chi-squared distribution
Scale-inv-$\chi^2(\nu_s, \tau_s^2)$ representing the number of facts provided by the source $\nu_s$ with variance $\tau_s^2$. The authors argue
that this distribution handles the effect of dataset size in the case of long-tail entities. The truth inference is performed
by leveraging prior knowledge from existing KGs.

**HYBRID** [82] tackles the TI task about knowledge on tail verticals and experiment the fusion by considering
two assumptions: single-truth and multi-truth. Before applying data fusion, they collect “evidences” on entities by
looking for whether a source contains the subject and object of the original triple. To do this, they use three types of
sources: knowledge bases (Freebase and Knowledge Vault), the Web, and query logs. The provenance information