dataesr/scientific-paragraphs-categorization huggingface dataset:
Model Validation Using the FRANK Benchmark The FRANK benchmark proposes a linguistically motivated typology of factual errors for fine-grained analysis of factuality in summarization systems: semantic frame errors, discourse errors and content verifiability errors. Semantic frames refer to the schematic representation of an event, relation, or state and a semantic frame error is an error that only involves participants of a single frame. Discourse errors extend beyond single semantic frames and consist of erroneous relations between different discourse segments. Content verifiability errors arise when information in the summary cannot be verified against the source document. The FRANK benchmark consists of humanannotated categorical error scores for 2250 model summaries. We use these annotations to compute partial correlation scores with human judgements for different models on different datasets and error types.
Partial Pearson correlation on different error types. Figure  2  shows the variation in partial Pearson correlation when flipping the labels of a specific error type. A higher positive bar indicates that the given type of error has high influence in the overall correlation. In other words, the factual consistency model performs well in detecting an error type if it has a high positive bar. We observe that DAE performs the best for content verifiability errors on both of the datasets, but gives unsatisfactory results for the other two error types. This is expected as DAE scores each dependancy arc independantly and thus cannot detect erros spanning across different arcs. BERTScore_Art and BartScore are both able to achieve positive results for discourse errors on XSUM, and for semantic frame errors on CNN/DM.
Examining Summarization Benchmark Datasets To question the validity of current summarization benchmark datasets, we first provide an overview of the context in which each dataset was created and the methodology employed in the construction procedures. In addition, we make use of the previously mentioned factual consistency models to evaluate the factuality of human reference summaries collected in each dataset. Our selection of summarization datasets is based on their popularity in recently published papers introducing new summarization systems. We believe that it is crucial to examine their validity as they play a fundamental role in both the development and evaluation of new systems. The most popular benchmark datasets according to our criteria are CNN/DM  (Hermann et al. , 2015)  and XSUM  (Narayan et al.
CNN/DM The CNN/DM dataset  (Hermann et al. , 2015)  was initially constructed as a Question Answering dataset composed of newswire articles in English and their corresponding highlights from the two platforms CNN and Daily Mail. Cheng and Lapata (2016)  later converted it into a summarization dataset by simply concatenating these highlights into summaries. It has now become the most broadly employed summarization dataset for the English language. However, the summaries formed from the concatenation of bullet points exhibit low degrees of abstraction and coherence, which are both highly desirable qualities for abstractive summarization systems. The dataset consists of 311,971 document-summary pairs. XSUM  Narayan et al. (2018)  create another large-scale dataset for abstractive summarization by crawling online articles from the BBC platform.
They take the first line of an article as the summary and the rest of the article as the source document. This method for annotating summaries guarantees high levels of abstraction but creates other issues as the first line of an article is often not written to be the summary. It might either include meta-information such as the author and publication date or serve as background introduction thus containing information never mentioned again in the rest of the article. XSUM contains 226,711 documentsummary pairs. XL-Sum XL-Sum  (Hasan et al. , 2021 ) is a recently introduced abstractive summarization dataset containing document-summary pairs also extracted from the BBC platform. The automatic annotation strategy of summaries is similar to that of XSUM. However, they find the first line of many articles to contain meta-information and thus annotate the bold paragraphs instead as the summaries.
Factual Consistency of Summarization Benchmarks We focus on evaluating the factual consistency of human reference summaries contained in each of the summarization benchmark datasets. We often assume that human references are "gold standards" but there is a lot to question in their validity considering how they are annotated. Here, we perform an analysis on the human references using the three factuality models introduced in Section 3. 1: BERTScore_Art, BARTScore and DAE. The results are shown in Figure  3. We remark that factuality scores produced by all three models rank CNN/DM as the most factual dataset by a large margin. This is due to its low level of abstraction and thus do not indicate its superiority as an abstractive summarization dataset. However, it is also worth pointing out that the improvement achieved betweeen XSUM and the recently created XL-SUM is not significant.
Introducing the SummFC Dataset The approach we choose to address the factuality problem in these summarization datasets is to filter the samples by their factual consistency scores obtained in Section 4. 2. We apply this idea to construct SummFC, the new Summarization benchmark dataset with improved Factual Consistency. In the remainder of this section, we first present our filtration methodology, followed by statistics of the SummFC dataset. We then introduce the evaluation metrics used in the comparison of summaries generated by models trained on the original benchmarks and models trained on SummFC. Finally, we perform experiments on the three datasets introduced in Section 4. 1 and observe advantages of the SummFC dataset.
Filtration Methodology In Figure  4 , we illustrate our pipeline for filtration and evaluation of the three datasets. The   filtration methodology is based on the factuality models BERTScore_Art, BARTScore and DAE. For each of the three datasets, we filter out the bottom 25% of document-summary pairs scored by each of the three models. In other words, we only keep the intersection between the top 75% of samples scored by each factuality model. This choice is made because the three factuality models are shown to complement each other in the detection of erroneous samples (see Section 3. 2). Experiments in Section 5. 4 show that datasets filtered by combined models outperform single model filtration with the same threshold.
The SummFC Dataset In Table  2 , we show statistics for the SummFC datasets. For all three datasets, the selection ratio of samples to retain is between 50% and 60%. In the context of our filtration methodology removing the lowest scored 25% of samples for each factuality model, this final selection ratio proves that there is a high degree of overlap between the samples different factuality models choose to filter out. We also report the average length of documents and summaries in each dataset. We compute length as the number of words (instead of tokens) included in the text. We might expect to achieve higher factual consistency scores for shorter summaries, as they  naturally contain less information. However, we show that our filtration methodology does not particularly favor samples with shorter reference summaries.
Factual Consistency Models For testing the improvement in factual consistency of the produced summaries, we employ the same factuality models as we use for filtration in Section 3. 1: BERTScore_Art, BARTScore and DAE. BLANC BLANC  (Vasilyev et al. , 2020 ) is a reference-free evaluation metric for summarization. It is defined as a measure of how well a summary helps an independent, pre-trained language model while it performs its language understanding task (masked token task) on a document. In other words, BLANC measures the informativeness of a summary in the context of document understanding. Here, we use the BLANC-help version, which uses the summary by concatenating it to each document sentence during inference. ROUGE-2 ROUGE-2  (Lin, 2004 ) is a reference-based text generation metric computing the bigram overlap between reference and candidate summaries.
Experiments and Results We fine-tune the pre-trained BART-base model  (Lewis et al. , 2020)  from the Transformers library  (Wolf et al. , 2020)  on the different summarization datasets. Since our goal is not to advance summarization systems but to compare different benchmark datasets, we do not experiment with hyperparameter tuning. For all parameters except batch size, we use the default settings in the Transformers library. Both the training and prediction are done on a NVIDIA TITAN V GPU with a batch size of 8. We present results for models fine-tuned on the original benchmarks and SummFC in Table  3. We also create a random selection baseline for each dataset by uniformly sampling a random subset of samples with equal size to the SummFC selection. We observe that models trained on the SummFC selection of XSUM and XLSUM achieve the highest scores for all reference-free evaluation metrics.
Although the selection procedure is only based on factual consistency, we see that the summarization systems have also improved in the informativeness aspect as measured by BLANC. For the referencebased metric ROUGE-2, models trained on SummFC obtain comparable scores to those trained on the original dataset and beat the random baseline by a large margin. It is also interesting to remark that for some of the factuality scores, even the random baseline achieves better performance than the full train set. This further confirms the conclusion that too much erroneous training data hinders factual consistency in summarization models. The results on CNN/DM are consistent with the other two datasets, except for BARTScore and BLANC. We believe that our filtration methodology is slightly less effective on CNN/DM due to the fact that this dataset manifests the lowest degree of factual inconsistency.
Another important reason is that the BART model used in BARTScore is also fine-tuned on CNN/DM, which makes it biased on this dataset. However, it is worth noting that SummFC is considerably smaller in size compared to the original ones. This means that using SummFC, we can achieve better results on nearly all quality aspects while reducing training time and lowering the need for computational resources. Figure  5  shows metric scores obtained when fine-tuning on filtered XLSUM datasets with single factuality models at different thresholds. We show scores for BLANC and ROUGE-2 because these two metrics were not used during filtration. We also create a baseline by randomly selecting samples with proportions equal to the tested thresholds. As the filtration criteria becomes stricter, BLANC scores increase while ROUGE-2 scores decrease.
Conclusion In this paper, we demonstrate that popular summarization datasets suffer from the lack of factual consistency and that summarization models trained on these datasets are not adequate for the task of abstractive summarization. We show that this problem can be solved by filtering the benchmark datasets with scores from factual consistency models. We propose a filtration methodology combining three stateof-the-art factual consistency models and introduce the SummFC dataset. SummFC is a unified Summarization benchmark consisting of Factually Consistent samples chosen from CNN/DM, XSUM and XLSUM. Experiments indicate that, in general, models trained on the smaller SummFC generate summaries with higher quality than models trained on the larger original datasets. Our findings suggest that more deliberate considerations should be made in the construction of benchmark datasets and that continuous revisions for the already existing ones are particularly necessary.
Limitations In this work, we restrict ourselves to the most popular summarization datasets. The three analyzed datasets share many similarities with each other and thus cannot account for diversity. They are all single-document singlereference English language summarization datasets in the news domain. Due to the limited sequence length accepted by Transformerbased factual consistency models, our filtration methodology cannot be generalized for longer document-summary pairs which are frequently present in other domains such as scientific articles and creative writing. The methodology also cannot be generalized for other languages. Factual consistency models such as DAE are trained on annotated datasets which only exist for the English language. This provides motivation for improving the generalization of factual consistency models.
Another limitation of our work is the lack of human evaluation. We prove the superiority of the SummFC dataset by evaluating the generated summaries with automatic metrics. Although these metrics achieve state-of-theart correlations with human annotated scores, they are still known to significantly differ from human judgements. While the factual consistency models that we employ also represent the current state-of-the art, it is far from guaranteed that they are able to identify all the factuality errors. It is only with human evaluation that we can provide a complete picture of where SummFC falls on the full dataset quality continuum. There is undoubtedly still more work to be done in continuing to refine datasets to actually measure summarization.
Recent developments in experimental methods using evolution and selection approaches have the potential to provide large datasets of protein sequences with different behavior in vitro or in a cellular setting  (3). We have previously reported one such approach, the tripartite β-lactamase assay (TPBLA)  (27, 28) -which has been used to enhance the thermodynamic stability of globular proteins  (29, 30) -to rank the aggregation propensity of variants of Aβ and IAPP  (28) , to screen for small-molecule inhibitors of aggregation  (28) , and to evolve aggregation-resistant proteins of relevance to biopharma  (27). In the TPBLA, a peptide or protein of interest is fused in-frame between the two domains of TEM-1 (Temoneira-1) β-lactamase (β-lac), and the fusion protein is expressed in the periplasm of E.
ACKNOWLEDGMENTS. We thank our research groups for helpful discussions and  Nasir Khan  for excellent technical support. S. E. R. , N. G. , R. M. M. , and Y. X. are funded by  Wellcome  ( 204963 ). We thank  Hugh Smith  for his help in the beginning of the project. We are also grateful to the  University of Leeds , the  Biological and Biotechnology Research Council (BBSRC)  ( BB/M012573/1 ), and  Wellcome  ( 094232 ) for funding for CD, X-ray, MS (mass spectrometry), and  EM Facilities. The Switch Laboratory  is supported by the  Flanders Institute for Biotechnology (VIB , grant  C0401 ),  KU Leuven , and the  Fund for Scientific Research Flanders (FWO ,  G053420N  and  S000722N ). B. I. I. acknowledges funding from the  Laboratory of Excellence in Research on Medication and Innovative Therapeutics  ( LER-MIT)  ( ANR-10-LABX-33 ).
Metagenomics analysis is now routinely used for clinical diagnosis in several diseases, and we need confidence in interpreting metagenomics analysis of microbiota. Particularly from the side of clinical microbiology, we consider that it would be a major milestone to further advance microbiota studies with an innovative and significant approach consisting of processing steps and quality assessment for interpreting metagenomics data used for diagnosis. Here, we propose a methodology for taxon identification and abundance assessment of shotgun sequencing data of microbes that are well fitted for clinical setup. Processing steps of quality controls have been developed in order (i) to avoid low-quality reads and sequences, (ii) to optimize abundance thresholds and profiles, (iii) to combine classifiers and reference databases for best classification of species and abundance profiles for both prokaryotic and eukaryotic sequences, and (iv) to introduce external positive control.
Second, the methodology also greatly varies with different technologies (short versus long reads) and multiple sequencing equipment, different approaches (targeted versus shotgun metagenomics), or different targets (microbiome versus mycobiome versus virome  [2]. Finally, bioinformatic analysis of millions of sequences is a key challenge, and the reproducibility of pipelines is essential. Ye and colleagues published a remarkable paper on the benchmarking of metagenomics tools for taxonomic classification  [3]. Their study examined the performance of 20 metagenomic classifiers using datasets by comparing the size and growth of reference databases, along with specific key metrics: precision-recall of the classifiers, abundance profile distances for 20 classifiers, and the computational requirements.
Among the DNA classifiers, Kraken and its derivate tools Bracken, KrakenUniq, and Kraken2 have several advantages. Indeed, they provide good performance metrics, they are very fast on large numbers of samples once the database has been loaded, and they allow for the creation and use of custom databases. Only Bracken is based on a probabilistic approach to generate the final abundance profiles. Among the protein-based classifiers, Kaiju is recommended by Ye et al. [3]  to access fast and efficient classification with minimum memory requirements compared to other classifiers. The development of metagenomic analysis pipelines also includes the recently published Sunbeam based on Kraken1 and SqueezeMeta, which represents a co-assembly procedure without filtering  [4, 5]. Taxonomic classifiers are still burdened by high numbers of false-positive calls at low abundance.
Dataset and Quality of Reads We used the simBA525 data set tested by Ye et al. [3]. This dataset contains reads randomly chosen from 525 bacterial/archaeal species. It is composed of short synthetic reads generated using ART with default settings  [6]. Before filtering and gene mapping, a robust quality review process included the kmer analysis of raw metagenome sequence reads. We used the k-mer analysis method introduced by Onate et al. [7]. Then, we selected Trimmomatic to identify and remove low-quality sequences and contaminants, as it is recognized as particularly efficient in various studies  [8] [9] [10].
Databases Direct taxonomic classification is useful for quantitative community profiling and identifying organisms with close relatives in the database  [11]. Among the 20 classifiers presented in the paper of Ye et al. , only Bracken employs a probabilistic approach to generate the final abundance profiles  [3]. Each classifier tool uses precompiled reference databases, which can differ widely. Kraken and Kaiju are distributed with precompiled reference databases similar to RefSeq (completely assembled and annotated reference genomes of archaea, bacteria, and viruses from the NCBI RefSeq database). Unlike Kraken, Kraken Uniq, and Kaiju, Kraken 2 includes additional bacterial databases such as 16S Greengenes, 16S Silva, and 16S RDP. Eukaryotic RefSeq contains only 191 of 1897 (10%) fungal genome assemblies.
Kaiju was the only classifier that included a fungal database (fungal sequences from the NCBI RefSeq database) and nr_euk database (as option -s nr additionally included proteins from fungi and microbial eukaryotes). We added fungal genomes of medical interest such as Candida parapsilosis GCA_004026285. 1, Candida krusei GCA_002166775, Candida tropicalis GCA_002864075, Aspergillus flavus GCF_000006275, Aspergillus nidulans GCF_000149205, Scedosporium apiospermum GCF_000732125, Scedosporium boydii GCA_002221725, Trichosporon asahii GCF_000293215, and Geotrichum capitatum GCA_000817185. Furthermore, plasmid sequences and mobile genetic elements (MGEs) in the RefSeq database can lead to incorrect taxonomic classification, being a major concern. Those sequences may be shared with different bacterial species and cannot be used as a discriminatory marker for bacterial taxa.
Precision and Recall Scores across All Abundance Thresholds To estimate a good value for a specific abundance threshold, we computed the precision/recall values for all possible thresholds and generated a precision/recall curve from this data (Figure  2 ). We observed that when decreasing the threshold, precision first decreased slowly and then dropped sharply below 500 reads per species. Therefore, we recommend a cutoff of 500 reads minimum per species. [3]. Each point in the curves represents the precision and recall score for a specific read abundance threshold, calculated on a simulated dataset. We observed a sharp decrease in precision when the threshold was below 500 reads per species, indicating many false-positive species with low abundance. The figure also shows the cutoff values for recall of 0.
Databases Kraken 2 allowed the use of both a standard database and custom databases. The standard Kraken 2 database contained NCBI taxonomic information, as well as the complete bacterial, archaeal, and viral genome sequences in RefSeq, the human genome, and a collection of known vectors (UniVec_Core). Other genomes were also added, but such genomes have to meet certain requirements, i. e. , sequences must be in FASTA format (multi-FASTA is allowed). When genomes met these requirements, each sequence was added to the database's genomic library using the -add-to-library switch. In combination with Kraken 2, Kaiju was cleverly complete with fungal sequences from the NCBI RefSeq database by including additional proteins from fungi and microbial eukaryotes as option-s nr.
Quality Review of Reads As previously described, Onate et al. showed that the normalized Shannon entropy (NSE) with a score between 0 and 1 is a good indicator of the diversity and quality of the metagenomic sample. We thus propose to use the following values to evaluate dataset quality prior to more extensive bioinformatics analysis: an NSE > 0. 96 denotes a good quality dataset; an NSE < 0. 93 denotes a low-quality dataset and an NSE in the range of (0. 93-0. 96) is considered as inconclusive.
Databases The rapid growth in the number of reference databases represents a fundamental challenge for the clinical interpretation of the results. This emphasizes the importance of selecting databases and appropriate methods for interpreting results. The most popular reference databases are RefSeq complete genomes (RefSeq CG) for microbial species as well as the BLAST nt and nr databases for high-quality nucleotide and protein sequences, respectively (50 and 200 million sequences). Other databases include SILVA for 16S rRNA, with 2 million sequences, and GenBank, with a large number of genomes and lower quality control standards. The RefSeq database contained all the viral genomes of medical interest listed by the Society of French Virology (Traité de virologie médicale, Thomas Mourez, Sonia Burrel, David Boutolleau et al.
2e éd. Paris: Société française de microbiologie; Société française de virologie, 2019). Current resources for fungal identification were added in order to improve the database. For all classifiers, it was crucial to add fungal genomes of medical interest, such as Candida parapsilosis, Candida krusei, Candida tropicalis, Aspergillus flavus, Aspergillus nidulans, Scedosporium apiospermum, Scedosporium boydii, Trichosporon asahii, and Geotrichum capitatum. The human genome was included in the default databases of Kaiju, Kraken, KrakenUniq, and Kraken 2 (GRCh38). Reads mapping of the human genome were removed after Kraken 2 analysis for accurate quantification of microbial species.