dataesr/scientific-paragraphs-categorization huggingface dataset:
2)], which encodes the 16S rRNA, essential to translating messenger RNA into proteins. After removing SNPs associated with recombination, SNP calls were split by primary cluster designations using a hierarchical approach, grouping primary clusters based on their nested structure, while filtering at a minimum allele frequency of 0. 10 to avoid the identification of relatively rare alleles that were not necessarily indicative of their respective population genomic cluster. High-impact SNPs were then identified using the program snpEff. A detailed account of all 62 high-impact SNPs identified (including their predicted effect) is presented in Table  S7. We also looked at clustered mutations in the same hierarchical manner, leading to the identification of 122 candidate genes potentially influencing selection (Table  S8 ).
anthracis persistence, biogeography and evolution. AMR has gained increased attention as a major threat to public health throughout the world  [48, 49]. By documenting AMR genes on a global scale, we can gain a better understanding of how biogeography and persistence are transforming the genomic constitution of dangerous pathogens at both regional and wider scales  [50, 51]. Based on our analysis of over 350 whole genomes, we have identified ten AMR genes present in B. anthracis isolates collected from over 35 countries, many with the different population clusters examined in this study. Five of these genes are commonplace and can be found in the majority of isolates examined, whereas the other five are comparatively rare across the dataset. Given the application of commonly used antibiotic drugs, such as penicillin, doxycycline and ciprofloxacin, to treat B.
anthracis infections, the regions where rare antibiotic-resistant gene isolates were sampled may benefit from monitoring, in order to document the persistence of these novel, resistant population clusters and modify antibiotic treatments for effectiveness  [52, 53]. The resistance gene bcII for example, which was found in only six samples, is known to hydrolyse a large number of penicillins (Table  1 ). Rarer antibiotic-resistant gene strains such as these may be indicative of a larger problem with antibiotic resistance in other dangerous pathogens as well, especially if the overuse of certain antibiotics is driving resistance in those regions where novel resistance genes reside  [54]. The influence of bacteriophage sequences on population genomic structure across the global dataset is less clear.
The strongest predictor of population genomic structure when examining the dataset in its entirety was the presence of the AMR gene vmlR, which was completely absent in primary cluster 5 [which was A. Br. 001-A. Br. 004 (Ames, Sterne, Aust94, V770) in the original classification system], the most genetically diverse population cluster examined in this study from which isolates were collected across Europe, Asia, Africa and the Americas. Interestingly, isolation source (host, environment or industry) was the second strongest predictor, suggesting that some strains of B. anthracis may be better suited to different environmental circumstances (or at least more readily cultured within them). Previous work that has examined population genomic structure has suggested that environmental growth outside of the host is possible  [28].
Given that this recombination event is rooted in a gene responsible for protein translation, these results support the hypothesis that phenotypic and functional traits for this cluster may be substantially different from the others. We examined genes that were identified two methods for pinpointing candidates for selection (high-impact SNPs and clustered mutations) and found that a range of functional differences may be driving population genomic structure. Primary cluster 1 (C Branch) exhibited premature stop codons in two genes, a DNA-binding response regulator and a stage 0 sporulation regulatory protein. If these premature stop codons are hindering this cluster's ability to produce proteins and influencing the timing and magnitude of sporulation, then this may indeed be why they are so underrepresented in the global dataset, and comparatively rare.
Primary cluster 2 (B Branch) exhibited a premature stop codon in the TPR domain protein. TPR proteins may act as scaffolds for the assembly of different multiprotein complexes  [60]. A premature stop codon in this sequence may be similarly affecting primary cluster 2's ability to persist and reproduce, leading to its similar rarity across the remainder of the global dataset (N=13/356). When primary clusters 5 and 6 are examined as a unit we see that the chlorohydrolase family protein exhibits a premature stop codon. Hydrolase proteins commonly perform as biochemical catalysts that use water to break a chemical bond, which typically results in dividing larger molecules into smaller molecules  [61]. If this protein lacks the ability to perform this function, isolates specific to this group may be functionally different from the other population groupings.
When evaluating the population genomic structure of B. anthracis in light of biogeography, AMR, phage diversity and candidate genes for selection, we find varying explanations for differences in population genomic structure. Nevertheless, it should be noted that in a mined dataset such as this, inaccuracy in metadata and/or sequencing has the potential to produce unintentional errors. In addition, our dataset is highly biased towards developed countries where whole genome sequencing technology is readily available and government support for such work is more abundant. Given the complex dispersal history of this notorious pathogen and the competing factors that ultimately sculpt its global genomic architecture, no single factor alone can be attributed to its modern genomic constitution. Despite these limitations we were able to determine the most influential factors consistent with differences and similarities among lineages using modern bioinformatic techniques.
B. The Wallinga and Teunis computation of Rt. The Wallinga-Teunis (4) method is also implemented in the EpiEstim package and widely considered as a reliable method to compute the case reproduction number, R c t , retrospectively  (11). Its formulas to estimate R c t at time t require the knowledge of it for t = 0, • • • , t + f. Starting from the original definitions of the authors, we give a mathematical proof that their method is actually computing Rt by the F1 form of the renewal equation. The method is based on the following estimation of the "relative likelihood, p k,l , that a case k has been infected by case l", p k,l = Φ(t k -t l ) n m=1,m =k Φ(t k -tm) where n represents the reported cases and t k is the time of 728 infection for the case k. Wallinga and Teunis define the case 729 reproduction number by 730 R l = k p k,l.
Introduction Language model pre-training has become a key component to improve performances on a majority of Natural Language Processing (NLP) tasks  (Wang et al. , 2019). Most of the recent competitive architectures  (Devlin et al. , 2019; Lan et al. , 2020; Liu et al. , 2019b; Radford et al. , 2018)  are based on the efficient transformer layer introduced in  Vaswani et al. (2017). BERT  (Devlin et al. , 2019)  is one of these architectures that has been widely adopted for comprehension and generation tasks. It is a multi-layer transformer network, pretrained with different self-supervised objectives. Numerous variations of transformer architectures have been proposed to improve this approach  (Lan et al. , 2020; Liu et al. , 2019b; Radford et al. , 2018). However, this type of process is only evaluated on tasks composed of relatively short input text, GLUE  (Wang et al.
Another approach adopted to deal with long documents or multi-document is to select a sub-sample of the input that is small enough for the transformer model. Most of the state-of-the-art pipelines on the multihop question answering dataset HotpotQA  (Yang et al. , 2018 ) use a first model to retrieve the relevant pieces of text before feeding them to a transformerbased architecture  (Fang et al. , 2019a; Tu et al. , 2019). We argue that these solutions are not feasible to deal with tasks that require a global understanding of long documents. An example is extractive summarization, where the decision for each sentence should be based on the information of the complete document. To address these challenges, we propose a simple adaptation of the multi-layer transformer architecture that can scale to long documents and benefit from pre-trained parameters with a relatively small length limitation.
Global BERT-based Transformer Architecture In this part, we briefly recall the transformer layer from  Vaswani et al. (2017)  and its integration in the BERT model  (Devlin et al. , 2019). Then we describe our modifications of this architecture that allow the model to read longer documents. Transformers: The transformer architecture, based on a sequence of transformer layers, has been initially introduced in  Vaswani et al. (2017). The key idea of this layer is to produce a contextualized representation of an input sequence of tokens. It is composed of the succession of a multi-head selfattention, a first normalizer, a feed-forward neural network, and a second normalizer. This model, which has originally been introduced for machine translation, has then been adopted for most natural language comprehension tasks.
Most of the successful approaches  (Devlin et al. , 2019; Liu et al. , 2019b; Lan et al. , 2020)  are composed of multiple stacked transformer layers. In the remainder, we denote by T the transformation corresponding to the th , 1 ≤ ≤ L, transformer layer (T is a function from R N ×h to R N ×h , where N denotes the length of the sequence and h the hidden dimension). BERT  (Devlin et al. , 2019 ) is a multi-layer transformer encoder pre-trained on large text corpora. Two BERT architectures have been proposed in  Devlin et al. (2019) : BERT BASE composed of 12 stacked transformer layers with hidden dimension of 768 (L = 12, h = 768) and BERT LARGE composed of 24 layers of hidden dimension 1024 (L = 24, h = 1024). For both architectures, the input length is limited to 512 WordPiece tokens and the pre-training includes two self-supervised tasks, namely masked language modeling and next sentence prediction.
For masked language modeling, 15% of all the WordPiece tokens of the input sequence are masked or corrupted, and the model is used to predict the original token with a crossentropy loss. For next sentence prediction, the model is trained as a classifier to predict if two sentences are contiguous or not. The pre-training procedure uses the BooksCorpus  (Zhu et al. , 2015)  and documents from English Wikipedia. It requires 4 days of optimization on 16 TPU chips for BERT BASE and 64 TPU chips for BERT LARGE.
Stacked Propagation Layers We propose a hierarchical structure that uses pretrained transformers to encode local text blocks that will be used to compute document level representations. The novel contribution of this work, depicted in Figure  1 , is to incorporate recurrent hierarchical modules between the different transformer layers and not only on top of the model, as proposed in  several recent works  (Fang et al. , 2019b; Zhang et al. , 2019b; Tu et al. , 2020). Because we construct and propagate document level information between the layers, global and local information are fused at every level of the architecture. The text blocks can be sentences, paragraphs, or sections. We experiment using sentences as blocks because it generally does not exceed the maximum length allowed by pre-trained models and because BERT has demonstrated to be well adapted to represent such sequences.
Experiments We evaluate our approach, which we refer to as GBT-EXTSUM (for 'Global BERT-based Transformer for Extractive Summarization'), in the context of extractive summarization, the goal of which being to identify and extract from a document the pieces of text that are the most important  (Kupiec et al. , 1995). We view this task as a sentencelevel classification problem where each sentence has to be labeled according to its belonging to the summary or not. To validate the effectiveness of our approach, we propose to test it on three summarization datasets, namely ArXiv, PubMed and CNN/DailyMail: Table  1  presents some statistics on these three datasets. As one can note, for the scientific articles, the average number of tokens in the documents to summarize is way beyond the capabilities of a standard transformer pre-trained with BERT.
Label Generation In order to train extractive summarizers, one needs annotations in the form of sentence-level binary labels. To compute such annotations, we follow the work of  Kedzie et al. (2018)  and label all sentences by greedily optimizing the ROUGE-1 score of the extracted summary against the gold-standard summary associated with each article. These labels are only used at training time, the evaluation of the extracted summaries being done against the gold-standard summaries provided in the datasets. Table  2 : Summarization results on PubMed and arXiv. Except for BERT-based approaches, for Reformer-Ext and for Longformer-Ext, which we have reimplemented, the results of the baselines are taken from their associated paper as well as from  Cohan et al. (2018). Bold results correspond to the best scores of extractive summarizers.
Baseline Models We compare our approach to several well known published methods described below. These methods include SumBasic  (Vanderwende et al. , 2007) , LexRank  (Erkan and Radev, 2004) , LSA (Steinberger and Jezek, 2004), Attn-Seq2Seq  (Nallapati et al. , 2016) , Pntr-Gen-Seq2Seq  (See et al. )  and Discourse-aware summarizer  (Cohan et al. , 2018). The results for these models are the ones reported in the paper  (Cohan et al. , 2018). We also report the results of Sent-CLF and Sent-PTR, which are hierarchical sentence pointer and classifier, TLM-I+E (G,M) a mixed extractive/generative transformer language model from Subramanian et al. (2019), BIGBIRD  (Zaheer et al. , 2020) , PEGASUS  (Zhang et al. , 2019a)  and DANCER  (Gidiotis and Tsoumakas, 2020)  which are three abstractive methods.
Lastly, we developed several baseline models based on BERT, Longformer  (Beltagy et al. , 2020)  and Reformer  (Kitaev et al. , 2020) : BERT Ranker: We used a BERT ranker, similar to  Nogueira and Cho (2019)  in which each sentence of the document is processed individually. We apply BERT on each sentence 1 and use a Sigmoid layer, the input of which consists of the 1 This is possible as no sentence exceeds BERT token limitation. [CLS] representation of the sentence, to model the probability of the sentence to be selected. BERTSUMEXT has been introduced in  Liu and Lapata (2019b). This model is an adaptation of BERT for extractive summarization. Because this model takes as input the concatenation of all the tokens of the document, it cannot scale to the arXiv and PubMed datasets. We propose two variants: the first one is to take as input only the first 800 tokens of the document, as suggested in the original paper.
This solution is displayed as BERTSUMEXT in Table  2. The second is to apply BERTSUMEXT per sliding windows on the original document and to use, as a token representation, its representation in the window that maximizes its surrounding context. We name this sliding window implementation BERTSUMEXT (SW) in Table  2. Longformer-Ext and Reformer-Ext: The Longformer and Reformer models were respectively introduced by  Beltagy et al. (2020)  and  Kitaev et al. (2020). They both propose an adaptation of the Transformer self-attention that scale to long sequences. We add the same classification head as the one used in our model on top of the contextualized representation of the first token of each sentence to label them as selected or not in the summary. We also present the Oracle extractive results as an upper bound as well as the Lead baseline (which respectively select the first 3, 6, 7 sentences for CNN/DailyMail, arXiv and PubMed datasets).
9, β 2 = 0. 999, no learning rate warmup and a linear decay of the learning rate. We describe implementation details of BERTSUMEXT, Longformer-Ext and Reformer-Ext baselines in the Supplementary Material, Appendix A. We used Trigram Blocking to avoid the repetition of trigrams in the extracted summaries as suggested in  Paulus et al. (2018). Given the extracted summary so far, we only added candidate sentences that had no overlapping trigram with the current summary. We limited the summary to 3 sentences for the CNN/DailyMail dataset, 6 sentences for arXiv, and 7 for PubMed.
Results Our main results are shown in Tables  2  and 3. On the arXiv and PubMed datasets, our model outperforms the baseline models on almost all of the reported metrics. Our approach manages to summarize long documents while preserving informativeness (evaluated by ROUGE-1) and fluency (evaluated by ROUGE-L) of the summaries. In addition   (Zhou et al. , 2018)  41. 59 19. 01 37. 98 SUMO  (Liu et al. , 2019a)  41. 00 18. 40 37. 20 TransformerExt  (Liu and Lapata, 2019b)  40. 90 18. 02 37. 17 MASK-LM global  (Chang et al. , 2019)  41. 2 19. 1 37. 6 PNBERT  (Zhong et al. , 2019)  42. 69 19. 60 38. 85 BERT-ext + RL  (Bae et al. , 2019)  42. 76 19. 87 39. 11 HIBERT M  (Zhang et al. , 2019b)  42. 37 19. 95 38. 83 BERTSUMEXT  (Liu and Lapata, 2019b)  43 to the previously published methods, our approach also improves over the BERT-based, Longformer-Ext and Reformer-Ext baselines we have developed.
Among them, BERTSUMEXT, which focuses on a truncated version of the document, is the less effective. As documents are significantly longer than the 800 tokens limitation of this model, this result is not surprising. The sliding window adaptation of this model, that allows it to scale to long documents, is the one that achieves results that are the most comparable to ours. Our approach still outperforms this adaptation, demonstrating that summaries require to propagate information beyond a single BERT window. On the CNN/DailyMail dataset, one can see that our model outperforms all the models that do not use pre-trained parameters. This includes several transformer-based and hierarchical models. However, while having comparable results, we do not achieve stronger performance than the current extractive state of the art from  Zhong et al.
(2020). This is not surprising as the majority of the CNN/DailyMail examples contains their oracle summary sentences in the first positions of the articles, as shown in the Supplementary Material, Appendix B. Lastly, we evaluate the impact of several elements of our proposed model in Table  4. We first study the influence of the underlying language model by considering both RoBERTa  (Liu et al. , 2019b)  and PEGASUS  (Zhang et al. , 2019a)  pre-trained models, respectively referred to as GBT-EXTSUM-RoBERTa and GBT-EXTSUM- PEGASUS. As one can see, the results show that BERT-base architecture performs best in terms of ROUGE scores on both arXiv and PubMed. One major difference between PEGA-SUS and BERT/RoBERTA pre-trained models is that BERT/RoBERTA are only encoders while PE-GASUS is a pre-trained encoder/decoder architecture.
28-the number of microsatellite repeats was estimated automatically using the genescan 672 software ( applied biosystems ) by the local southern method with a size marker of gs500 tamra ( applied biosystems ). 22-polymerase chain reaction ( pcr ) was performed in a reaction mixture with a total volume of 12. 5 l containing pcr buffer , genomic dna , 0. 2 mm dinucleotide triphosphates ( dntps ) , 0. 5 m primers , and 0. 35 u taq polymerase. Table  5 : An example of summary produced by our method compared to the gold summary and one produced by BERTSUMEXT (SW). With a red scale, we highlight the sentences with the highest ROUGE score when evaluated against the abstract. We show in the margin the position of the extracted sentence in the document. This document  (Kamio et al. , 2009)  is 78 sentences long.
Conclusion In this paper, we have introduced a novel transformer-based model for long document summarization based on propagation layers that spread information between multiple transformer windows. This model preserves the architecture of commonly used pre-trained language models, thus allowing the transfer of parameters. An evaluation, conducted on top of the BERT model in the context of an extractive summarization task, further revealed its effectiveness in dealing with long documents compared to other adaptations of BERT and previously proposed models. In the future, we plan to adapt our model to other tasks that require understanding long documents, as question-answering and document-scale machine translation.
A Baselines: Implementation Details BERTSUMEXT: For all experiments with BERTSUMEXT, we started with the original implementation 3 and adapted the code to build the sliding windows version. This implementation leverage bert-base-uncased pre-trained model and its associated hyperparameters. We use windows of width 800 with an overlap of 300 tokens between two following windows. If a sentence is in multiple windows, we select its [CLS] representation in the window that maximizes the number of surrounding tokens. We finetune the model for 5 epochs using Adam optimizer with an initial learning rate of 1 × 10 -5 , β 1 = 0. 9, β 2 = 0. 999.
Reformer-Ext: We started from the HuggingFace implementation of Reformer to build Reformer-Ext baseline. We use a Reformer configuration composed of six layers of attention. We use Locality-Sensitive Hashing Attention with 128 buckets on the input sequence and Local Selfattention on chunks of 64 tokens. We use hidden sates of dimension 256, a feed-forward layer of dimension 512, and 12 attention heads in Transformer encoders. We train this model for 5 epochs using Adam optimizer with an initial learning rate of 1 × 10 -5 , β 1 = 0. 9, β 2 = 0. 999, no learning rate warmup and a linear decay of the 3 https://github. com/nlpyang/PreSumm 4 https://github. com/huggingface/ transformers 5 https://github. com/allenai/longformer learning rate.
B Datasets Statistics Figure  4  presents the distribution of the document lengths in arXiv, PubMed and CNN/DailyMail, after tokenization with pretrained BERT-base tokenizer. It also provides the histograms of the position of the [CLS] tokens of the Oracle sentences in input documents. One can see that the three datasets contain an important number of documents longer than 512 tokens, the standard length limitation of pre-trained language models. However, one can also notice that CNN/DailyMail contains a large part of its Oracle sentences within this first window of 512 tokens. As a consequence, a model that is not able to "read" beyond this limitation is not penalized. It is also a reason why Lead baseline is quite strong on this dataset. On the contrary, on arXiv and PubMed, one can see that a large part of Oracle sentences occur beyond this 512 windows.
This explains why models capable of reading long sequences are required to achieve good results on these datasets. C PubMed Summaries GOLD aim. to investigate incidental adrenal enlargement clinical characteristics and functional status and analyze functional lesion risk factors. materials and methods. this retrospective study included 578 patients with adrenal imaging features showing enlargement. incidental adrenal enlargement cases ( 78 ) were considered eligible. demographics , functional diagnosis , adrenal imaging features , and concomitant diseases were analyzed. results. the number of adrenal enlargements and proportion of incidental adrenal enlargement increased each year. mean patient age was 50. 32 years. thirty -nine cases had unilateral enlargement on the left side and 3 on the right side ; 36 had bilateral enlargement.