dataesr/scientific-paragraphs-categorization huggingface dataset:
As an example, BIGSI  [3]  relies on Bloom filters with a high false-positive rate, e. g. 25% false-positive rate per kmer query. At constant memory usage, adding the abundance information would yield an extremely high false-positive rate. As such, methods storing abundances mostly rely on compression by clustering abundance with neighbouring k-mers or across datasets, as Reindeer  [10]  or Counting de Bruijn graphs  [6]. These methods do not rely on counting AMQ, but rather on exact data structures. In this paper, we do not propose a novel counting AMQ, but rather a wrapper to improve any existing counting AMQ, like a counting Bloom Filter. The method we introduce is called fimpera. It generalises one of our previous contribution  [11]. In short, fimpera splits every k-mer into s-mers (with k ≥ s > 0) and then associates the abundance of a k-mer to its constituent s-mers in a counting AMQ.
Preliminary definitions A k-mer is a word of length k over an alphabet Σ. Given a sequence S, |S| denotes the length of S. In the current framework, we consider a dataset as composed of one sequence or a multisets of sequences. Given a dataset D, D k denotes multiset of k-mers extracted from D. We denote the abundance of a k-mer d (the number of time d appears) in D k by abundance(D k , d). We consider that a k-mer is "present" (resp. "absent") in D if abundance(D k , d) > 0 (resp. abundance(D k , d) = 0). A counting AMQ data structure represents a multiset of elements D k. It can be queried with any element d; the query's response on an counting AMQ, noted n, is always either correct or overestimated, i. e. n ≥ abundance(D k , d). If n = abundance(D k , d), the counting AMQ reported the correct abundance, otherwise it reported an overestimation.
Indexation overview At indexation time, fimpera takes a file of counted kmers, typically extracted from a genomic sequence dataset, and splits each k-mer to be indexed into its k -s + 1 s-mers (k ≥ s). Each s-mer is then stored in a cAMQ along with its s abundance. The s abundance of a s-mer is the maximum of the abundance of the k-mers containing this s-mer. We explain this choice in the following. In the following, we set z = k -s, hence z ≥ 0. Query overview The query of fimpera consists of a set of sequences. For each sequence S, fimpera extract s-mers, which are then queried against the cAMQ, and the abundance of any k-mer of S is computed as the minimum of s abundance of its s-mers. By default, fimpera prints each input sequence along with the abundance of every of its consecutive k-mer.
For instance, with z=3 (which is a recommended and default value), a counting AMQ having a false positive rate of 25%, the probability of false-positive rate with fimpera for that setting is ≈ 0. 04%. The fimpera approach may generate a novel kind of false-positives. A queried k-mer, absent from the indexed dataset, may be composed of s-mers, all existing in this indexed set. Querying such k-mer with fimpera returns a non-zero abundance, so generating a false-positive, that we call a "construction false-positive". This new kind of false-positive call is specific to the fimpera approach. Overview of overestimations of fimpera's query To overestimate the abundance of a queried k-mer, overestimations are required to happen on the abundance of every s-mer of that k-mer. The more s-mer per k-mer, the more s-mer abundance overestimations need to happen to overestimate a k-mer abundance.
Indexing with fimpera As stated in Section 2. 2, fimpera's indexation is a two-step process: k-mers' abundances are computed from the input dataset (e. g. using KMC  [7] ); s-mers from these k-mers are stored in a counting AMQ together with their s abundance. The s abundance of a s-mer is formally defined as the maximal abundance of the indexed k-mers in which this s-mer occurs. Note that the s abundance of a s-mer α is lower or equal to the abundance of α in the input dataset. For instance consider a s-mer α that occurs in two k-mers respectively with an abundance of one and two. Then, the abundance of α is three (= 1 + 2), while the s abundance of α is two (= max(1, 2)). Storing the s abundance of α instead of its abundance, enables to lower the abundance overestimations, as it avoids to accumulate the abundances of distinct k-mers it belongs to.
Implementation of fimpera An implementation of fimpera is available at https://github. com/lrobidou/ fimpera. This implementation is specialised for genomic data (i. e. with an alphabet consisting of A, T, C, G) and uses a counting Bloom filter as cAMQ. A template mechanism allows the use of any other cAMQ provided by the user. In the genomic context, queries consist of fasta or fastq files (gzipped or not), and an option is provided to index and query canonical k-mers only, i. e. the lexicographic minimum between each k-mer and its reverse complements. Options include the k and z values, the size of the filter, and b, the number of bits per abundance count. As b has a major impact on the final size of the data structure, it is recommended to use low b values (say b ≤ 5). This limits the maximal stored abundance value to 2 b.
Experimental setup To the best of our knowledge, no other tool focuses on reducing the false positive rate of existing cAMQ, thus we compare fimpera results applied on a counting Bloom filter indexing s-mers with the original counting Bloom filter results indexing k-mers. Both methods are using a single hash function. We propose results on biological marine metagenomic data. Parameters used are the default ones: k = 31, size of filter of 3. 48 × 10 9 bits, as discussed in section 3. 3, using b = 5 bits per abundance count, and abundances are stored as their ⌊log 2 ⌋ values. We use the default z = 3 parameter (unless otherwise stated). A list of commands for reproducing the results is available here: https:// github. com/lrobidou/fimpera/blob/paper/paper_companion/Readme. md along with a step-by-step explanation of the output.
Metagenomic dataset We used two fastq files from the TARA ocean metagenomic dataset to show advantages offered by fimpera on metagenomic samples. The index was computed from the 2. 38 × 10 8 31 -mers present at least twice in an arctic station (accession number ERR1726642) and the query sample was the first 3×10 6 reads from a sample in another arctic station (accession number ERR4691696). Canonical k-mers were considered for this experiment.
Quality of results As shown in Table  3. 8, the false positive rate decreases with regard to z and stays low for a wide range of z values (at least from 3 to 9). When using an extreme z value, for instance s = 20, the false positive rate is increased up to almost 100%. With z = 20, as we use k = 31, the size of the s-mers is s = 11. When indexing as little as few hundred millions characters, each 11-mer has a great chance to appears by chance in the indexed dataset (it does not occurs with a probability of 10 -11 when indexing a hundred million characters on an alphabet of size four). This quasi-random existence of all s-mers generates a huge amount of construction false positives, as seen in the last column. This has also an impact on the running time that nearly doubles, certainly because all queried s-mers are positives, annihilating the s-mer skipping optimisation.
Conclusion We presented fimpera, a novel computational method to reduce the false positive rate and increase the precision of any counting Approximate Membership Query data structure with no modification of the original data structure. This reduction is obtained without any memory overhead, with no modification of the original data structure, and even with a slight improvement over the query computation time. Our results showed that when applied on top of a counting Bloom Filter, fimpera enabled to yields about 45 times fewer false-positive calls than when querying directly a counting Bloom Filter of identical size. Moreover, using fimpera, abundance errors were slightly less frequent on true positive calls, and finally, those abundance errors were on average 1. 07 apart from the ground truth with fimpera while they are on average 1.
33 apart from the ground truth with the original cBF. Independently from parameters of the used cAMQ, fimpera requires to set up a unique parameter, z. Fortunately results are highly robust with the choice of z, unless extreme values are chosen. Future work will include a formal analysis of the theoretical limits on the choice of z usage ranges. We provide a C++ implementation of fimpera which enabled us to validate the approach. This implementation can also be used as a stand-alone tool for indexing and querying genomic datasets, and it can be tuned with user-defined parameters and ranges of abundances. The provided github project also proposes all necessary instructions and links to genomic data to reproduce the results.
For instance, they are not wellsuited for drawing comparisons across datasets  [5]. More importantly, they only provide a partial picture of KGEM performance  [5]. Indeed, LP can lead to nonsensical triples, such as (BarackObama,isFatherOf,USA), being predicted as highly plausible facts, although they violate constraints on the domain and range of relations  [23, 73]. KGEMs with such issues may nevertheless reach a satisfying performance in terms of rank-based metrics. Few works propose to go beyond the mere traditional quantitative performance of KGEMs and address their ability to capture the semantics of the original KG, e. g. , domain and range constraints, hierarchy of classes  [22, 40, 49]. According to Berrendorf et al. [5] , this would give a more complete picture of the performance of a KGEM.
To do so, we build on Sem@K, a semantic-oriented metric that we previously introduced  [20, 21]  In  [20] , Sem@K was specifically defined for the recommendation task which was seen as predicting tails for a unique target relation. Sem@K was then extended in  [21]  to the more generic LP task, where not only tails but also heads are corrupted and all relations are considered. In the present work, we deepen the study of the semantic awareness of KGEMs by proposing different versions of Sem@K that take into account the different characteristics of KGs (e. g. , hierarchy of types). Moreover, the semantic awareness of a wider range of KGEMs is analyzed -especially convolutional models. Likewise, a broader array of KGs is used, in order to benchmark the semantic capabilities of KGEMs on mainstream LP datasets.
their performance in terms of rank-based metrics along training epochs. We show that a trade-off may exist for most KGEMs. our study supports the view that agnostic KGEMs are quickly able to infer the semantics of KG entities and relations. The remainder of the paper is structured as follows. Related work is presented in Section 2. Section 3 outlines the main motivations for assessing the semantic awareness of KGEMs. Section 4 subsequently presents Sem@K, the semantic-oriented metric that fulfills this purpose. Sem@K comes in different flavors based on the typology of the datasets at hand and the intended use cases. In Section 5, we detail the datasets and KGEMs used in this work, before presenting the experimental findings in Section 6. A thorough discussion is provided in Section 7. Lastly, Section 8 outlines future research directions.
MR alleviates this concern as it does not consider any threshold K. Therefore, MR allows to compare KGEM performance on the same dataset. Nonetheless, MR is sensitive to the number of KG entities (see Eq. (  2 ))  [5] : a MR of 10 indicates very good performance if the set of entities is in the thousands, but it would indicate poor performance if the set of entities is much more restricted. Therefore, MR does not allow comparisons across datasets. Recent works recommend using adjusted version of the aforementioned metrics. The Adjusted Mean Rank (AMR) proposed in  [1]  compares the mean rank against the expected mean rank under a model with random scores. In  [5] , Berrendorf et al. transform the AMR to define the Adjusted Mean Rank Index (AMRI) bounded in the [0, 1] interval. This way, a value of 1 indicates an optimal performance of the model.
Motivating example To motivate the use of Sem@K to evaluate KGEM quality, this section builds upon a minimalist example which is representative of the issue encountered while benchmarking the performance of several KGEM on the same dataset. As depicted in Fig. 2 , two KGEMs that have been trained on the whole training set are tested on a batch of test triples. These KGEMs are referred to as Model A and Model B. Without loss of generality, it is assumed that the test set only comprises the three triples shown in Fig. 2. For the sake of clarity, only the tail prediction pass and the top-5 ranked candidate entities are depicted in Fig. 2. It should be noted that the performance of both models are strictly equal in terms of MR, MRR and Hits@K with K 5: MR=8/3, Hits@1=1/3, Hits@3=2/3 and Hits@5=1.
MRR can only be computed knowing the total number of entities in the KG but two models having the same MR on the same dataset have de facto the same MRR as well. Distinguishing between these two models by relying solely on traditional rank-based metrics is not possible. One might even draw the misleading conclusion that these models are equally good. But they are not: Model A gives high scores to semantically valid entities with regard to the range of relations, while Model B semantic awareness is very low. In other words, Model B does not capture the semantic profile of relations well. This case in point illustrates the need for using semantic-oriented metrics alongside common rank-based metrics, so as to better assess the overall quality of KGEMs.
Datasets In order to draw reliable and general conclusions, a broad range of KGs are used in our experiments. They have been chosen due to their mainstream adoption in recent research works around KGEMs for LP and the fact that they have different characteristics (e. g. entities, relations, classes, presence of a class hierarchy). In this section, the schema-defined and schemaless KGs used in the experiments are detailed. Note that in our experiments, all the schema-defined KGs come with a class hierarchy inherited from either Freebase  [6] , DBpedia  [2] , YAGO  [60] , or schema. org. To meet requirements for evaluating KGEMs w. r. t. Sem@K (i. e. classes instantiated by entities, domain and range for relations), when necessary, subsets of schema-defined KGs are used, as explained in Section 5.
Schema-defined KGs The statistics of the datasets FB15K237-ET, DB93K, YAGO3-37K and YAGO4-19K used in our experiments are provided in Table  2. As discussed in Section 4. 2, to create an experimental evaluation setting as unbiased and flawless as possible, the schema-defined KGs used in the experiments are filtered to keep typed entities only. This way, Sem@K is calculated under the CWA. FB15K237-ET derives from FB15K  [7]  -a dataset extracted from the cross-domain Freebase KG  [6]. In these experiments, we do not use FB15K, as it has been noted that this dataset suffers from test leakage, i. e. , a large number of test triples can be obtained by simply inverting the position of the head and tail in the train triples  [65]. The later introduced FB15K237 dataset  [65]  is a subset of the original FB15K without these inverse relations.
Contrary to the initial version of DB100K, the latter version is schema-defined: entities are properly typed and most relations have a domain and/or a range. This second version is considered in the following experiments. However, some inconsistencies were found in the dataset. Some DBpedia entities only instantiate Wikidata 4  or schema. org 5  classes, while instantiation of classes from the DBpedia ontology actually exist. Moreover, some entities are only partially typed. It must also be noted that domains and ranges of relations have been extracted from DBpedia more than two years ago. DBpedia is a communautary and open-source project: DBpedia classification system relies on human curation, which sometimes implies a lack of coverage for some resources and updates for others. Consequently we associated all relations of DB100K to their most up-to-date domains and ranges.
6  Similarly for entities, we associated all entities to their most up-to-date classes. Finally, we removed all untyped entities as well as validation and test relations having less than 10 observed entities, so as not to unfairly penalize Sem@K results in the validation and test phases. YAGO3-37K derives from the schema-defined YAGO39K dataset  [37]  extracted from the cross-domain YAGO3 KG  [60]. Compared to the original YAGO39K, in our experiments only typed entities are kept. In addition, relations having less than 10 observed heads or tails in the training set are discarded from the validation and test splits, for the same reason that keeping them would not reflect the actual Sem@K values. It should be noted that in the YAGO3 ontology, most relations have very generic domains and ranges which are very close to the root of the ontology hierarchy.
To produce a more challenging evaluation setting of the models' semantic awareness, a subset of hard relations was identified and only validation and test triples whose relation belongs to this subset are kept. The resulting dataset is named YAGO3-37K. We built YAGO4-19K with several other subsets of the YAGO4 knowledge graph  [63]. Similarly to other datasets, we focused on relations with a defined domain and range, and more than 10 triples to constitute the validation and test sets. We purposedly favored difficult relations to feature in the validation and test sets. To enrich the training set, additional relations were added based on a manual selection. Selected relations in validation and test sets as well as additional relations in the training set can be found on the GitHub repository of the dataset.
Extensional KGs Another range of datasets used in these experiments do not come with an ontological schema. In particular, this means relations do not have a clearly-defined domain or range. Although Codex-S and Codex-M are based on the Wikidata schema which does possess property constraints linking subject types to value type constraints, we limit ourselves to KGs that represent this information with rdfs:domain and rdfs:range predicates. Consequently, in this work, we only report Sem@K[ext] results for Codex-S and Codex-M. The statistics of the datasets Codex-S, Codex-M and WN18RR used in our experiments are provided in Table  3. WN18RR  [13]  originates from WN18, which is a subset of the WordNet KG  [39]. As for FB15K, Toutanova et al. [65]  reported a huge test leakage in the original WN18 dataset.
More specifically, 94% of the train triples have inverse relations that are linked to test triples. Dettmers et al. [13]  remove all inverse relations to propose WN18RR. In WN18RR, entities are nouns, verbs, and adjectives. Relations such as hypernym and derivationally related from hold between observed entities. Such relations are not linked to any rdfs:domain or rdfs:range predicates. Besides, relations such as derivationally related from can contain nouns, verbs or adjectives as both head or tail. It follows that it is not possible to infer any expected entity type for relations -in this case the word qualifier. This is why WN18RR is used in the extensional setting. Codex-S  [56]  and Codex-M  [56]  are datasets extracted from Wikidata and Wikipedia. They cover a wider scope and purposely contain harder facts than most KGs  [56].
Consequently, these datasets prove to be more challenging for the link prediction task. Compared to WN18RR, Codex-S and Codex-M contain entity types, relation descriptions and Wikipedia page extracts. Nonetheless, Wikidata does not contain rdfs:domain or rdfs:range predicates. The property constraints present in Wikidata are harder to manipulate than the rdfs:domain and rdfs:range clauses found in DBpedia. As such, in our experiments Codex-S and Codex-M are used in the extensional setting. Codex-S and Codex-M initially come with already generated hard negatives. In our experiments, we do not directly use these provided negative triples. Instead, we use the same Uniform Random Negative Sampling schema as for other datasets.
Finally, the following scoring function assesses the plausibility of a given triple (h, r, t): f (h, r, t) = concat g [e h , e r , e t ] * ω • w (14) It has been claimed that the concatenation of a set of feature maps generated by convolution should increase the learning ability of latent features compared to ConvE  [25]. However, recent works point out the evaluation procedure used in the original implementation of ConvKB  [58, 62] , which may result in overly optimistic results on the benchmark datasets FB15K237 and WN18RR. Nonetheless, due to the popularity of this model, we choose to include it in our experiments. R-GCN  [57]  extends the idea of applying graph convolutional networks (GCNs) to multi-relational data. R-GCN operates on local graph neighborhoods and applies a convolution operation to the neighbors of each entity.
Implementation details and hyperparameters For the sake of comparisons, MRR, Hits@K and Sem@K all need to rely on the same code implementation. More specifically, for R-GCN  7  and CompGCN, 8  existing implementations were reused and Sem@K values were calculated on the trained models. Other KGEMs were implemented in PyTorch. To avoid time-consuming hyperparameter tuning, we took inspiration from the hyperparameters provided by LibKGE 9  for Codex-S, Codex-M, WN18RR and FB15K237. However, LibKGE does not benchmark all the datasets and models used in our experiments. For such models, we stick to the hyperparameters provided by the original authors, when available. For the models with no reported best hyperparameters, as well as for the remaining datasets used in the experimentsi. e. DB93K, YAGO3-37K and YAGO4-19K -different combinations of hyperparameters were manually tried.
We first trained our KGEMs for 1,000 epochs, then noticed the best achieved results were found around epoch 400 or below. Consequently, we stick to a maximum of 400 epochs of training as in LibKGE (except R-GCN which is trained during 4,000 epochs due to lower convergence to the best achieved results). For each positive triple in the training set, one corresponding negative triple is generated. To ensure fair comparisons between our models, we stick with Uniform Random Negative Sampling  [7]. The chosen hyperparameters leading to the best performance on the validation dataset are provided in Appendix (see Appendix A, Table  7 ). Recall that the objective of this work is to perform a fair and insightful assessment of the semantic awareness of KGEMs. As such, the intended purpose was not to reach optimal performance in terms of rank-based metrics.
range) of a given relation, while semantic matching models might sometimes be better at predicting entities already observed in the domain (resp. range) of a given relation (e. g. DistMult reaches very high Sem@K[ext] values on Codex-S, as evidenced in Fig. 7a ). In cases when translational and semantic matching models provide similar results in terms of rank-based metrics, the nature of the dataset at hand -whether it is schema-defined or schemaless -might thus strongly influence the final choice of a KGEM. Table  5  Results on YAGO3-37K. Bold fonts and gray cells denote the best achieved results and the worst achieved results among the models reported in the table, respectively. Full results are available in Appendix B, Table  8  Rank-based Sem@K[base] Sem@K[wup] Sem@K[ext] Model Family Model MRR H@1 H@3 H@10 S@1 S@3 S@10 S@1 S@3 S@10 S@1 S@3 S@10   Interestingly, CompGCN which is by far the most recent and sophisticated model used in our experiments, outperforms all the other models in terms of semantic awareness as, with very limited exceptions, it is the best in terms of Sem@K[base], Sem@K[ext] and Sem@K[wup].